Systems and methods for 3D surface measurements

ABSTRACT

Certain embodiments are directed to 3D surface measurement systems and methods configured to direct engineered illumination of beams in one or more ray bundles to N illumination directions incident a sample surface. The systems and methods can measure a map of surface normals, a depth map, and a map of surface properties using intensity images captured while the engineered illumination is directed to the sample surface.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/364,320, titled “Systems and Methods for 3D Surface Microgeometryand Reflectance Properties Measurement” and filed on Jul. 20, 2016 andto U.S. Provisional Patent Application No. 62/511,964, titled “Systemsand Methods for 3D Surface Microgeometry and Reflectance PropertiesMeasurement” and filed on May 27, 2017, each of which is herebyincorporated by reference in its entirety and for all purposes.

FIELD

Certain embodiments described herein are generally related tomeasurement systems, and more particularly, to methods and systems for3D surface measurements (3DSM methods and systems) used to measuresurface microgeometry and/or surface properties as may be implemented,for example, in manufacturing for defect detection and quality control,as well as in other applications such as computer graphics.

BACKGROUND

The measurement of the 3D surface microgeometry (also referred to astopography) with high depth resolution has proven to be very difficultand is the focus of active research in the fields of 3D metrology andcomputer vision. In the field of 3D metrology, a non-contact opticalapproach is of particular interest driven by customer requirements forgreater simplicity and higher speed. Some conventional techniques thatcan be used to measure surface depth with nanometer to micron-scaledepth resolution include white light interferometer, confocalmicroscope, and focus variation. Conventional instruments that use thesetechniques generally have, however, a narrow field-of-view, shortworking distance, and shallow depth-of-field since they use highmagnification microscope objectives. These conventional instruments alsotend to be very expensive due to their use of sophisticated optical andmechanical components. Moreover, the devices based on interferometry arecommonly sensitive to the ambient environment, such as vibration. Also,most of these techniques are designed to take surface topographymeasurements, and are unable to capture reflectance properties such asdiffuse reflection and specular reflection. An example of a white lightinterferometer can be found in De Groot, P., “Principles of interferencemicroscopy for the measurement of surface topography,” Advances inOptics and Photonics, vol. 7, no. 1, pp. 1-65 (2015), which is herebyincorporated by reference for this example. An example of focusvariation device can be found in Matilla, A., et al., Three-dimensionalmeasurements with a novel technique combination of confocal and focusvariation with a simultaneous scan,” Proc. SPIE, pp. 98900B-11, (2016),which is hereby incorporated by reference for this example.

In the field of computer vision, some research has been done on 3Dsurface measurement with somewhat simpler setups. The photometric stereotechnique is a method to reconstruct surface normal and depth bycapturing multiple images under different lighting directions, but thismethod assumes the scene to be Lambertian (diffusive). An example of amethod to reconstruct surface normal and depth by capturing multipleimages under different lighting directions can be found in Basri, D., etal., “Photometric stereo with general, unknown lighting,” InternationalJournal of Computer Vision, vol. 72, no. 3, pp. 239-257 (2007), which ishereby incorporated by reference for this example. In real world most ofthe objects are not purely diffusive, and can be specular or hybrid.Recently new techniques have been explored to address objects that arenot diffusive. Some methods have been proposed to address microgeometrymeasurement on specular surfaces. Examples of such methods can be foundin Chen, T., et al, Mesostructure from specularity,” Proc. CVPR, pp.1825-1832 (2006) and Francken, Y., et al., “High quality mesostructureacquisition using specularities,” in Proc. CVPR, p. 1-7 (2007), whichare hereby incorporated by reference for these examples. These proposedmethods either use hand-moved point light source or structuredillumination to reconstruct depth from specularity, but their approachescan only capture images with limited number of lighting directions andtherefore may result in insufficient samplings of the reflectance field.In another example, an object is pressed into an elastomer skin toremove the specular reflection of the object, and a photometric stereotechnique is used to estimate the surface normal based on reflectionfrom the covered skin. An example can be found in U.S. PatentPublication 20130033595, titled “HIGH-RESOLUTION SURFACE MEASUREMENTSYSTEMS AND METHODS,” by Adelson, H., and Johnson, Micah K., which ishereby incorporated by reference for this example. However, when usingthis approach, the true reflection from the sampled object is blocked bythe elastomer skin, and information about the color or other reflectanceproperties is lost.

In the field of computer graphics, some research has been done on thesimultaneous acquisition of surface geometry and reflectance propertiesfor computer graphics rendering. For example, a photometric stereotechnique for objects with spatially varying bidirectional reflectancedistribution function (BRDF) can be found in Goldman, D. B., et al.“Shape and spatially-varying BRDFs from photometric stereo,” IEEE Trans.PAMI, vol. 32, no. 6, pp. 1060-1071 (2010), which is hereby incorporatedby reference in its entirety. In this example, multiple images arecaptured under different lighting directions, and then the shape andreflectance parameters are reconstructed from a BRDF model forrendering. However, this method requires that the camera andillumination be at far distance from the sample in order to fulfill theorthographic assumption, and the method is unable to capture detailedmicrogeometry of the surface due to limited number of illuminationdirections. In addition, a reflectometry technique has been proposed forsurface normal, height, diffuse and specular reflectance parametersmeasurement, but this method requires a linear light source be movedacross the object surface. An example of such a reflectometry system canbe found in U.S. Pat. No. 6,919,962, titled “Reflectometry apparatus andmethod,” which is hereby incorporated by reference for this example. Aspecular object scanner for measuring reflectance properties of objectshas also been proposed, but this scanner requires an arc-shaped lightsource to be rotated around the object. An example of such a specularobject scanner can be found in U.S. patent application Ser. No.14/212,751 titled “Specular object scanner for measuring reflectanceproperties of objects,” which is hereby incorporated by reference forthis example. In addition, a 12-light hemispherical dome for capturingdetailed microgeometry of skin texture has been developed, which mayimprove the realism of facial synthesis. An example of such a system isfound in Graham, P., et al., “Measurement-based synthesis of facialmicrogeometry,” in Computer Graphics Forum, vol. 32, no. 2, pt. 3. WileyOnline Library, pp. 335-344 (2013), which is hereby incorporated byreference for this example. However, in this example, the 12-lighthemispherical dome has limited angular samplings above the surface, andmay not work well for specular objects that are smoother. To acquiremuch denser samplings, the example discusses using a light stage, butthe light stage is large and some of the lights may be occluded by thecamera.

In these conventional techniques, the camera and multiple illuminationsare placed at a far distance in order to fulfill the orthographicassumptions. Such a setup typically introduces a large form factor, andsome of the illuminations may be occluded by the camera. Because thecamera is not able to capture images of the surface closely, it isdifficult to achieve micron-level depth resolution and accuracy. Theseconventional techniques use a limited number of extended light sources,and are typically arranged with large angular sampling steps due to theconstraints in their physical dimensions. This may result ininsufficient samplings of the reflectance field, and bias the surfacenormal estimation for specular surfaces that are relatively smoother.Moreover, typically some extended illuminations were placed at obliqueangles introducing severe shadow effects. Most of these conventionaltechniques are unable to measure the reflectance properties of thesurface.

SUMMARY

Certain embodiments pertain to methods and systems for 3D surfacemeasurements.

Certain embodiments are directed to a 3D surface measurement systemcomprising an engineered illumination system configured to provide atleast one ray bundle to N illumination directions incident a surface ofa sample being imaged. Each ray bundle comprises illumination beams ofvarious intensities. The 3D surface measurement system further comprisesa camera with an imaging lens and at least one sensor configured tocapture intensity images at the N illumination directions based on lightreceived from the illuminated sample. In addition, the 3D surfacemeasurement system comprises a controller configured to executeinstructions to determine a sensor response at each sensor pixel fromthe intensity images at the N illumination directions, match the sensorresponse at each sensor pixel to one of a plurality of predeterminedsensor responses to determine a surface normal at each sensor pixel, andconstruct a map of surface normals of the surface by combining thedetermined surface normal of all the sensor pixels of the at least onesensor.

Certain embodiments are directed to a 3D surface measurement method thatreceives a plurality of intensity images of a sample in a signal from atleast one sensor of a camera. The plurality of intensity images havebeen captured at a plurality of exposure times at each illuminationdirection of N illumination directions of one or more ray bundles,wherein each ray bundle comprises illumination beams of variousintensities. The method also determines a sensor response at each sensorpixel from the intensity images at the N illumination directions. Inaddition, the method matches a sensor response at each sensor pixel toone of a plurality of predetermined sensor responses to determine asurface normal at each sensor pixel and constructs a map of surfacenormals by combining the determined surface normal of all the sensorpixels of the at least one sensors.

Certain embodiments are directed to a 3D surface measurement method thatengineers at least one ray bundle to N illumination directions incidenta surface a sample being imaged, each ray bundle comprising illuminationbeams of various intensities. The method also captures, using at leastone sensor of a camera, wherein the intensity images are captured at aplurality of exposure times at each illumination direction based onlight from the illuminated sample. In addition, the method communicatesthe intensity images to one or more processors. The one or moreprocessors determine a sensor response at each sensor pixel from theintensity images at the N illumination directions, match a sensorresponse at each sensor pixel to one of a plurality of predeterminedsensor responses to determine a surface normal at each sensor pixel, andconstruct a map of surface normals by combining the determined surfacenormal of all the sensor pixels of the at least one sensor.

Certain embodiments are directed to a 3D surface measurement method forcalibrating a 3D surface measurement system. The method comprisesengineering at least one ray bundle to N illumination directionsincident a surface of a mirror or a chrome sphere, each ray bundlecomprising illumination beams of various intensities. The method furthercomprises capturing, using at least one sensor of a camera, intensityimages at each illumination direction based on light reflected from themirror or the chrome sphere. The method further comprises determining acalibration offset and the N illumination directions based on theintensity images.

These and other features are described in more detail below withreference to the associated drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a simplified schematic drawing depicting a singlemicrosurface of a sample having a surface normal N, according to oneimplementation

FIG. 1B shows a schematic illustration of a series of eighty-one (81)predetermined sensor responses corresponding to eighty-one (81) surfacenormal orientations of a 3DSM system, according to an implementation.

FIG. 2 is a simplified block diagram depicting components of a 3DSMsystem, according to various implementations.

FIG. 3A is a graph with a first transmission profile of a first diffuserand a second transmission profile of a second diffuser, both profileshaving distributions that are Gaussian or Gaussian-like, according toembodiments.

FIG. 3B is a graph with a linear gradient profile, according to anembodiment.

FIG. 3C is a graph of a random distribution profile, according to anembodiment.

FIG. 4 is a schematic drawing depicting a side view of components of a3DSM system during a data acquisition phase of a 3DSM method, accordingto an embodiment.

FIG. 5 is a schematic drawing depicting a side view of components of a3DSM system during a data acquisition phase of a 3DSM method, accordingto an embodiment.

FIGS. 6A-6C are schematic diagrams depicting three operations of a dataacquisition phase of a 3DSM method performed by a 3DSM system, accordingto various embodiments.

FIG. 6D is a schematic diagram depicting a general approach taken duringan analysis phase performed by the 3DSM system of FIGS. 6A-6C, accordingto various embodiments.

FIG. 7 is a schematic drawing depicting a 3DSM system configured togenerate engineered illumination using collimated illumination and adiffuser, according to an embodiment.

FIG. 8 is a sensor response image captured without engineeredillumination.

FIG. 9 is a sensor response image captured with engineered illuminationprovided by the 3DSM system shown in FIG. 7, according to an embodiment.

FIG. 10A is a schematic drawing depicting components of a 3DSM systemincluding a combination filter, according to an embodiment.

FIG. 10B is an illustration of a transmittance profile of a combinationfilter, according to an implementation.

FIG. 10C is an illustration of a transmittance profile of a combinationfilter, according to an implementation.

FIG. 10D an illustration of a transmittance profile of a combinationfilter with a linear gradient, according to an implementation.

FIGS. 11A-11B are schematic diagrams depicting two operations of a dataacquisition phase of a 3DSM method performed by a 3DSM system configuredfor engineered illumination direction manipulation, according to anembodiment.

FIG. 12 is a schematic diagram depicting an operation of a dataacquisition phase of a 3DSM method performed by a 3DSM system, accordingto an embodiment.

FIG. 13 is a schematic drawing depicting components of a 3DSM systemincluding additional illumination devices, according to an embodiment.

FIGS. 14A-14B are schematic diagrams depicting components of a 3DSMsystem configured for illumination direction manipulation, according toan embodiment.

FIGS. 15A-15C are schematic diagrams depicting a calibration processusing a mirror as a reference object performed by a 3DSM system,according to an embodiment.

FIGS. 16A-16B are illustrations of an offset of incident beams resultingfrom the calibration process depicted in FIGS. 15A-15C, according to animplementation.

FIGS. 17A and 17B are illustrations depicting a system alignment processusing the results of the calibration process depicted in FIGS. 15A-15C,according to an implementation.

FIGS. 18A-18C are schematic diagrams depicting three operations of asystem calibration process performed by a 3DSM system including achromosphere array, according to an embodiment.

FIGS. 19A-19B are images of a single chromosphere of a chromospherearray illuminated at two different illumination directions during acalibration process, according to an embodiment.

FIGS. 20A and 20B are illustrations of the calibrated zenith angles andazimuth angles for 169 different illumination directions using thecalibration process depicted in FIGS. 17A-17C, according to animplementation.

FIGS. 21A-21C is a schematic diagram depicting a 3DSM systemimplementing a calibration process using a combination of a mirror and asingle chromosphere, in accordance with an embodiment.

FIG. 22 depicts a series of pre-determined sensor responses of a singlesensor pixel, according to an embodiment.

FIG. 23 is a schematic representation depicting the operation ofmatching a measured sensor response for a single sensor pixelcorresponding to a surface pixel to one of a series of pre-determinedsensor responses, according to an embodiment.

FIG. 24A is a photograph of a leather sample showing an area beinganalyzed by a 3DSM system, according to an embodiment.

FIG. 24B is an illustration of a measured depth map of the area of theleather surface of FIG. 24A as measured by the 3DSM system.

FIG. 24C is an illustration of a measured surface normal map of the areaof the leather surface of FIG. 24A as measured by the 3DSM system.

FIG. 24D is an illustration of a measured diffuse map of the area of theleather surface of FIG. 24A as measured by the 3DSM system.

FIG. 24E is an illustration of a specular map of the area of the leathersurface of FIG. 24A as measured the 3DSM system.

FIG. 24F is an illustration of a surface roughness map of the area ofthe leather surface of FIG. 24A as measured the 3DSM system.

FIG. 25A is a photograph of a coin.

FIG. 25B is a 3D rendering using measured depth map of the coin shown inFIG. 25A, according to an embodiment.

FIG. 25C is graph having a measured depth curve measured along the blackline of the coin as shown in FIG. 25A, according to an embodiment.

FIGS. 26A-26D are renderings based on microgeometry and reflectanceproperties measured by an 3DSM system, according to an implementation.

FIGS. 27A-27C are schematic diagrams depicting operations of a dataacquisition phase implemented by a 3DSM system configured to controlwavelength and polarization of the engineered illumination, according toan embodiment.

FIGS. 28A-28C are schematic diagrams depicting three operations of adata acquisition phase implemented by a 3DSM system that controlswavelength and polarization of the reflected light from the surface ofthe sample, in accordance with an embodiment.

FIGS. 29A-29C are schematic diagrams depicting three operations of adata acquisition phase implemented by a 3DSM system that controlswavelength and polarization of both the engineered illumination and thereflected light from the surface of the sample, in accordance with anembodiment.

FIG. 30 is a schematic diagram illustrating components of a 3DSM systemconfigured to control wavelength and polarization of both the engineeredillumination and the reflected light from the surface of the sample, inaccordance with an embodiment.

FIGS. 31A-31C are schematic diagrams depicting operations of a dataacquisition phase implemented by a 3DSM system configured to control themotion of the sample surface, according to an embodiment.

FIGS. 32A-32C are schematic diagrams depicting operations of a dataacquisition phase implemented by a 3DSM system configured to control themotion of the engineered illumination and the camera, according to anembodiment.

FIG. 33 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface emissivity properties using a 3DSM system configured to controlthe motion of the sample surface, according to an embodiment.

FIG. 34 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface emissivity properties using a 3DSM system configured to controlthe motion of the engineered illumination and the camera, according toan embodiment.

FIG. 35 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface emissivity properties using a 3DSM system configured to controlthe wavelength and/or the polarization of the emitted light from thesample surface, according to an embodiment.

FIG. 36 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface translucency properties using a 3DSM system configured tocontrol the motion of a translucent sample, according to an embodiment.

FIG. 37 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface translucency properties using a 3DSM system configured tocontrol the motion of a backlight, according to an embodiment.

FIG. 38 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface translucency properties using a 3DSM system configured tocontrol the engineered illumination and the camera, according to anembodiment.

FIG. 39 is a schematic diagram depicting operations of a dataacquisition phase and an analysis phase of a 3DSM method that obtainssurface translucency properties using a 3DSM system configured tocontrol the wavelength and/or polarization of both the backlight andlight transmitted from the surface of the sample, according to anembodiment.

FIG. 40 is a schematic diagram depicting the 3DSM system of FIGS. 11A-Band FIG. 12 that is configured to switch the engineered illumination toan off state, according to an implementation.

FIG. 41 is a schematic drawing depicting a data acquisition operation ofa 3DSM system, according to an embodiment.

FIG. 42 is a flowchart of operations of a 3DSM method, according tovarious implementations.

FIG. 43 is a flowchart depicting sub-operations of an operation of the3DSM method of FIG. 42.

FIG. 44 is a flowchart depicting sub-operations of an operation of the3DSM method of FIG. 42.

FIG. 45 is a block diagram depicting inputs and outputs of the renderingoperation, according to an embodiment.

FIG. 46A is an illustration of diffuse reflection properties of asynthetic leather surface as measured by a 3DSM system, according to anembodiment.

FIG. 46B is an illustration of specular reflection properties of thesynthetic leather surface as measured by the 3DSM system of FIG. 46A,according to an embodiment.

FIG. 46C is an illustration of glossiness properties of the syntheticleather surface as measured by the 3DSM system of FIG. 46A, according toan embodiment.

FIG. 46D is an illustration of a measured base color that describes thecombined color appearance of both diffuse reflection and specularreflection of the synthetic leather surface as measured by the 3DSMsystem of FIG. 46A, according to an embodiment.

FIG. 46E is an illustration of a measured metallic map that describesthe dielectric and metallic properties of the synthetic leather surfaceas measured by the 3DSM system of FIG. 46A, according to an embodiment.

FIG. 46F is an illustration of roughness properties measured by the 3DSMsystem of FIG. 46A, according to an embodiment.

FIG. 46G is an illustration of a normal map of the synthetic leathersurface as measured by the 3DSM system of FIG. 46A, according to anembodiment.

FIG. 46H is an illustration of measured depth map of the syntheticleather surface as measured by the 3DSM system of FIG. 46A, according toan embodiment.

FIG. 46I is a physically-based rendering of the synthetic leathersurface as measured by the 3DSM system of FIG. 46A, according to anembodiment.

FIG. 47 is a block diagram depicting operations of a 3DSM method thatincludes a defect detection process, according to an embodiment.

FIG. 48 is a block diagram depicting operations of a 3DSM method thatincludes a defect detection process, according to an embodiment.

FIG. 49A is an illustration of specular reflection properties of an LCDpanel that is contaminated with dirt and fiber measured by a 3DSM systemconfigured for defect detection, according to an embodiment.

FIG. 49B is an illustration of diffuse reflection properties of the LCDpanel with dirt and fiber as measured by the 3DSM system of FIG. 49A,according to an embodiment.

FIG. 49C is an illustration of surface roughness properties of the LCDpanel with dirt and fiber as measured by the 3DSM system of FIG. 49A,according to an embodiment.

FIG. 49D is an illustration of a measured normal map of the LCD panelwith dirt and fiber as measured by the 3DSM system of FIG. 49A,according to an embodiment.

FIG. 49E is an illustration of a measured depth map of the LCD panelwith dirt and fiber as measured by the 3DSM system of FIG. 49A,according to an embodiment.

FIG. 49F is an illustration of surface defect visualization operationusing the measured parameters illustrated in FIGS. 49A-49E using a 3DSMmethod including defect detection implemented by the 3DSM system,according to an embodiment.

FIG. 50A is an illustration of measured specular reflection propertiesof an LCD panel with scratches and pits as measured by a 3DSM systemconfigured for defect detection, according to an embodiment.

FIG. 50B is an illustration of measured diffuse reflection properties ofan LCD panel with scratches and pits as measured by the 3DSM system ofFIG. 50A, according to an embodiment.

FIG. 50C is an illustration of measured surface roughness properties ofan LCD panel with scratches and pits as measured by the 3DSM system ofFIG. 50A, according to an embodiment.

FIG. 50D is an illustration of measured normal map of an LCD panel withscratches and pits as measured by the 3DSM system of FIG. 50A, accordingto an embodiment.

FIG. 50E is an illustration of measured depth map of the LCD panel withscratches and pits as measured by the 3DSM system of FIG. 50A, accordingto an embodiment.

FIG. 50F is an illustration of surface defect detection using parametersmeasured by the 3DSM system of FIG. 50A, according to an embodiment.

FIG. 51 is an illustration depicting a measured brightness map of an LCDpanel with a white spot as measured by a surface defect detectionprocess in a 3DSM method implemented by a 3DSM system, according to anembodiment.

FIG. 52 is an illustration depicting a measured uniformity map of theLCD panel with non-uniformity as measured by a surface defect detectionprocess in a 3DSM method implemented by the 3DSM system, according to anembodiment.

DETAILED DESCRIPTION

The following description is directed to certain implementations for thepurposes of describing various aspects of this disclosure. However, aperson having ordinary skill in the art will readily recognize that theteachings herein can be applied in a multitude of different ways. Thus,the teachings are not intended to be limited to the implementationsdepicted solely in the Figures, but instead have wide applicability aswill be readily apparent to one having ordinary skill in the art.

Embodiments pertain to methods and systems for 3D surface measurements(3DSM methods and systems) including measurements of microgeometry andsurface properties such as reflectance properties, emissivityproperties, translucency properties, etc. The 3DSM systems implementmethods that can be used to simultaneously acquire surface geometry andproperties. In various embodiments, a 3DSM system comprises anengineered illumination system that generates and manipulates incidentdirections of engineered illumination to different points along asurface of a sample to N different illumination directions. Theengineered illumination has a known distribution such as a Gaussiandistribution, a pattern, etc. During data acquisition, light from thesurface is captured by one or more sensors of a digital camera for the Ndifferent illumination directions. The 3DSM system captures a measuredsensor response for each sensor pixel for the N different illuminationdirections. The 3DSM system determines a surface normal for each of thesensor pixels, for example, by matching the measured sensor response tothe best fitting of a group of pre-determined sensor responses. Eachpre-determined sensor response corresponds to a surface normal. Thepre-determined sensor responses for different surface normals can becalculated based on the known distribution of the engineeredillumination at different illumination directions. During an analysisphase, the 3DSM system determines a surface normal at each point of thesurface by matching the measured sensor response to a pre-determinedsensor response. The 3DSM system can estimate a depth map from thesurface normals at the different points across the surface. The 3DSMsystem can also use the map of surface normals to determine surfaceproperties such as reflectance properties e.g. diffuse albedo, specularalbedo and roughness, estimated, for example, based on a BRDF model.

I. Introduction to 3DSM Systems and Methods

Most real-world surfaces are not flat, and can be approximated as aseries of flat microsurfaces facing various directions. Each of theseflat microsurfaces has a surface normal vector (also referred to hereinas a “surface normal”). A two-dimensional (2D) grid of surface normalsat points across a three-dimensional surface corresponds to the contoursof the surface. A depth map (topographical) of the surface and varioussurface properties such as, e.g., reflective properties, can bedetermined from the 2D grid of surface normals. Examples of techniquesthat can be used to determine a depth map and surface properties fromsurface normals can be found in U.S. Patent Publication 20130033595,titled “HIGH-RESOLUTION SURFACE MEASUREMENT SYSTEMS AND METHODS,” byAdelson, H., and in Johnson, M. K., et al., “Shape and spatially-varyingBRDFs from photometric stereo,” IEEE Trans. PAMI, vol. 32, no. 6, pp.1060-1071 (2010).

In certain implementations, a 3DSM system comprises an engineeredillumination system configured to direct a ray bundle havingillumination beams of varying intensities to points (also referred toherein as “pixels”) along the surface of a sample. The illuminationbeams of the ray bundle have a known engineered intensity distribution.In many cases, the engineered illumination system includes a motioncontrol device configured to rotate and/or translate one or more systemcomponents to cause the rotation of the ray bundle relative to thesample. The ray bundle is rotated in two orthogonal directions to directthe ray bundle to different illumination directions incident the samplewhile a camera measures the intensity distribution of light propagatedfrom the illuminated sample. The intensity distribution measured duringan exposure time is referred to herein as an “intensity image” or simplyas an “image.” The intensity images can be used to generate the 2D gridof surface normals of the sample.

In one implementation, either the sample is manipulated(rotated/translated) or the engineered illumination and camera aremanipulated together to scan the engineered illumination over the sampleduring data acquisition. The 3DSM system analyzes the captured images tomeasure maps of surface normals, depths, and surface properties thesample using the images captured. During data acquisition of a largearea sample, the engineered illumination is scanned over a region of thelarge area sample and then scanned over another region, and so on, untilthe entire sample is scanned. The 3DSM system can stitch together themeasured maps of the surface normals, depths, and surface properties forthe different regions of a large area sample to generate maps of theentire surface. This approach may have one or more technical advantages.One technical advantage may be the ability to measure over a large areasurface. Another technical advantage is that the measured maps of thesurface normals, depths, and surface properties for the differentregions of a large area sample can be stitched together, which providesfor high spatial resolution in the measured maps of the entire surface.Another technical advantage is that the 3DSM system can more preciselycontrol collimated illumination to a smaller region of the sample thanwould be possible with a large area surface.

FIG. 1A is a simplified schematic drawing of a single microsurface of asample having a surface normal N, according to an embodiment. In thissimplified scenario, a ray bundle of three (3) beams L1, L2, and L3 ofdifferent intensities is shown incident a surface point P of themicrosurface. As shown, only the L1 beam is reflected (shown as areflected beam R) back to a camera and a sensor of the camera measuresan intensity value of the L1 beam. The L2 and L3 beams are not reflectedback to the camera. Since the three (3) beams L1, L2, and L3 areengineered with a particular known intensity distribution for variousoutput angles, the sensor reading corresponds to the known intensityvalue of the L1 beam. In this way, the 3DSM system can use the sensorreading to identify the L1 beam as reflected from the microsurface. Theangle of the illumination beam L1 is also known based on the intensitydistribution and the configuration of any optical elements propagatingillumination to the surface. The 3DSM system can determine the surfacenormal of the microsurface based on the known angle of the identifiedillumination beam L1 and the orientation of the sensor.

In certain scenarios, the engineered illumination comprises a ray bundleof illumination beams where some beams have the same or substantiallysimilar intensities. In addition or alternatively, components of thesystem may have various transmission properties and sensor may havelimited dynamic range. In these cases, a sensor reading of a particularintensity value may not be unique to a single illumination beam andcould correspond to multiple beams having similar output power. Forexample, if beam L1 and beam L3 shown in FIG. 1A had the same power, thesensor reading based on receiving beam L1 would correspond to both ofthe illumination beams L1 and L3. In order to determine the beam beingreflected from the microsurface and estimate the surface normal, certain3DSM systems described herein take sensor readings at differentillumination directions of the ray bundle at each surface point. In somecases, the 3DSM systems include a motion control device that rotates oneor more system components to rotate the ray bundle together in twoorthogonal directions to direct the ray bundle to a sequence of Nillumination directions. For a particular 3DSM system, each surfacenormal has a unique sensor response for N illumination directions basedon rotating the ray bundle in two orthogonal directions. Each measuredsensor response is a pattern of intensity values measured at a sensorpixel for the different illumination directions. For example, a sensorresponse pattern may have four hundred (400) intensity values associatedwith four hundred (400) illumination directions based on 20 rotationalangles along an x-direction and 20 rotational angles along a y-directionorthogonal to the x-direction. In various implementations, the 3DSMsystem compares the measured sensor response to predetermined sensorresponses. The 3DSM system determines a predetermined sensor responsethat matches most closely to the measured sensor response to estimatethe surface normal at each pixel.

FIG. 1B shows a schematic illustration of a series of eighty-one (81)predetermined sensor responses corresponding to eighty-one (81) surfacenormal orientations of a 3DSM system, according to an implementation. Inthis example, each surface normal has a unique 20×20 sensor responsepattern of normalized intensity for four hundred (400) rotationalpositions corresponding to twenty (20) rotational angles along a y-axis(rotation angle Y) and twenty (20) rotational angles along an x-axis(rotation angle X) orthogonal to the y-axis.

II. 3DSM Systems

FIG. 2 is a simplified block diagram of a 3DSM system 10, according tovarious implementations. The 3DSM system 10 comprises a controller 20having a processor 22 and a memory 24 in electrical communication withthe processor 22. The processor 22 is configured to control functions ofthe 3DSM system 10 by executing instructions stored in memory 24. Theprocessor 22 may also retrieve data stored in memory 24 and/or storedata to memory 24. The 3DSM system 10 further comprises an engineeredillumination system 30 having one or more illumination devices 34, oneor more optical elements 36, and a motion control device 32. Eachillumination device 34 includes one or more light sources. Theillumination device(s) 34 and optical element(s) 36 are configured toprovide engineered illumination to the sample during operation. The 3DSMsystem 10 further optionally (denoted by the dashed line) comprises asample platform 40 for receiving the sample and camera(s) 60 with one ormore lenses and sensors configured to receive light from the sampleduring operation and to capture intensity measurements. The motioncontrol device 32 is in communication optionally (denoted by the dashedline) with one or more of the illumination device(s) 34, the opticalelement(s) 36, the camera(s) 60, and the sample platform 40 in order tocontrol their movement (e.g., translation/rotation) to rotated theengineered illumination relative to the sample to N rotationalpositions. For example, the motion control device 32 may include a motorcoupled to a system component to rotate and/or translate the componentso that the engineered illumination is rotated to different rotationalpositions over a sequence of sample times. The controller 20 is inelectrical communication with the illumination device(s) 34 and thecamera(s) 60 to send control signals to control functions of thesecomponents and to receive data, such as intensity measurements from thecamera(s) 60. For example, the controller 20 may send control signals tothe camera(s) 60 and the motion control device 32 to synchronizeexposure times of the camera(s) 60 to be at the same time that themotion control device 32 is holding system components in a particularposition to provide engineered illumination to a surface at a particularrotational position.

The 3DSM system 10 optionally (denoted by dashed line) further comprisesa communication interface 72 and a display 70 in communication with thecommunication interface 72. The controller 20 is configured orconfigurable to output raw data, processed data such as surfacemeasurements or renderings, and/or other data over the communicationinterface 72 for display on the display 70. In another embodiment, the3DSM system 10 may further comprise one or more additional communicationinterfaces and/or a computing device in communication with acommunication interface. In addition and optionally, the 3DSM system 10may further comprise an external memory device in communication with acommunication interface for storage of data to the external memorydevice, and/or a communication interface in communication with a userinterface for receiving input from an operator of the 3DSM system 10.The described electrical communication between system components may beable to provide power and/or communicate data. The electricalcommunication between system components of various implementations maybe in wired form and/or wireless form.

In various implementations, the 3DSM system includes an engineeredillumination system comprising one or more illumination devices, one ormore optical elements, and a motion control device. The engineeredillumination system is designed to generate and provide for the relativerotation of the engineered illumination in two orthogonal directionswith respect to the surface of a sample. Engineered illuminationgenerally refers to one or more ray bundles where each ray bundle iscomprised of multiple illumination beams. The illumination beams of eachray bundle are at different angles. The power of the illumination beamsat different output angles can be, for example, a linear, a triangular,a curved, or a random distribution. According to one aspect, the powerof the illumination beams at different angles is based on a Gaussiandistribution or a distribution similar to a Gaussian distribution. Inanother aspect, the power of the illumination beams at different angleshas a linear gradient. In yet another aspect, the power of theillumination beams at different angles is random.

In various implementations, a 3DSM system implements engineeredillumination with illumination beams having power for different outputangles in the form of a Gaussian or Gaussian-like distribution. In thesecases, the engineered illumination system generally comprises opticalelements that include a collimator and a diffuser. The collimator isconfigured to collimate illumination from one or more light sources. Thediffuser is configured to receive collimated illumination and generatethe one or more ray bundles. Each ray bundle has an infinite number ofillumination beams with output power for different output angles in aGaussian or Gaussian-like distribution. Examples of suitable diffusersare commercially available. For example, a suitable commercial diffuseris the DG20-1500 manufactured by Thorlabs.

FIG. 3A is a graph illustrating a first transmission profile 310 througha first diffuser and a second transmission profile 320 through a seconddiffuser, according to embodiments. These are examples of typicaldiffuser transmission profiles. Each of the transmission profilesexhibits has a Gaussian-like distribution of output power ofillumination beams for different output angles (i.e. angles from therespective diffusers). The transmission profiles 310, 320 are graphs ofthe distribution of normalized output power values of the infinitenumber of illumination beams for different output angles in degrees fromthe respective diffuser. The transmission profiles 310, 320 areillustrated to point out three beams along each profile at the outputangles of −7 degrees, 0 degrees, and 6 degrees. The first transmissionprofile 310 has a first beam 336 having a normalized power of about 0.17at −7 degrees, a second beam 332 having a normalized power of about 0.98at 0 degrees, and a third beam 338 having a normalized power of about0.20 at 6 degrees. The second transmission profile 320 has a first beam330 having a normalized power of about 0.6 at −7 degrees, a second beam332 having a normalized power of about 0.98 at 0 degrees, and a thirdbeam 334 having a normalized power of about 0.67 at 6 degrees. Anexample of commercially-available diffusers that can provide anon-Gaussian intensity distribution are Engineered Diffusers™ fromThorlabs RPC Photonics.

In another aspect of the 3DSM system, the engineered illuminationincludes one or more ray bundles with each ray bundle havingillumination beams with power varying based on output angle according toa linear gradient distribution. FIG. 3B is a graph illustrating a lineargradient profile 340 of the distribution of illumination beams of a raybundle for different output angles, according to an embodiment. Thelinear gradient profile 340 includes normalized power of an infinitenumber of beams at different output angles. The linear gradient profile340 has a first beam 342 having a normalized power of 0.5 at −10degrees, a second beam 344 having a normalized power of about 0.3 at 0degrees, and a third beam 346 having a normalized intensity of about 0.2at 10 degrees. In this example, the gradient is decreasing linearly asoutput angle increases. In other cases, the gradient is increasinglinearly.

In another aspect of the 3DSM system, the engineered illuminationincludes one or more ray bundles with each ray bundle havingillumination beams with power varying based on output angle according toa random distribution. FIG. 3C is a graph illustrating a profile 350 ofa random distribution of beams, according to an implementation. Theprofile 350 includes normalized power of the illumination beams atdifferent output angles. The profile 350 has a first beam 352 having anormalized power of about 0.65 at −10 degrees, a second beam 354 havinga normalized power of about 0.95 at 0 degrees, and a third beam 356having a normalized power of about 0.3 at 10 degrees.

FIG. 4 is a schematic drawing showing a side view of components of a3DSM system 400 during a time period of operation at which the camera(s)460 takes multiple exposures at different settings, according to oneembodiment. The 3DSM system 400 comprises an engineered illuminationsystem 430, a sample platform 440, and a camera(s) 460. A sample 480with a surface 482 is shown disposed on the sample platform 440. The3DSM system 400 includes an x-axis parallel to the surface 480 at thepoint L, a z-axis normal to the surface 480 at point L, and a y-axis(not shown) orthogonal to the x-axis and the z-axis. At this instant intime, the engineered illumination system 440 is shown providing a raybundle of six (6) beams of various intensities at different incidenceangles to a surface point L at the surface 482 of the sample 480. Theray bundle is directed along a 0 degree illumination direction normal tothe surface 482. The rotational position of the ray bundle is at 0degrees rotation along the x-axis and 0 degrees rotation along they-axis. It would be understood that although six (6) beams areillustrated for simplicity, an infinite number of beams are incidentsurface point L. The engineered illumination system 430 comprises amotion control device 432, optical element(s) 436, and illuminationdevice(s) 434. The motion control device 432 is coupled to the opticalelement(s) 436 and to the camera(s) 460 to be able to translate and/orrotate one or both of these system components to various positions torotate the ray bundle of six (6) illumination beams to differentrotational positions at different sample times during operation. Themotion control device 432 is optionally (denoted by dashed line) becoupled to the illumination device(s) 434 to be able to translate and/orrotate the illumination device(s) 434 during operation. For example, theray bundle may be rotated incrementally along the x-axis and/or alongthe z-axis. At each rotational position, the one or more sensor(s) ofthe camera(s) 460 takes intensity measurements (images) of multipleexposures of light propagated from the illuminated sample 480.

FIG. 5 is a schematic drawing showing a side view of components of a3DSM system 500 during a time period of operation at which a cameratakes multiple exposures, according to one embodiment. The 3DSM system500 comprises an engineered illumination system 530 and a sampleplatform 540. A sample 580 with a surface 582 is shown disposed on thesample platform 540. At this instant in time, the engineeredillumination system 530 is shown providing three (3) ray bundlesincluding a first ray bundle of three (3) beams L1, L2, and L3 ofvarious intensities at different incidence angles to a surface point Lat the surface 582 of the sample 580, a second ray bundle of three (3)beams M1, M2, and M3 of various intensities at different incidenceangles to a surface point M, and a third ray bundle of three (3) beamsN1, N2, and N3 of various intensities at different incidence angles to asurface point N. Each of the ray bundles is directed along a 0 degreeillumination direction normal to the surface 582. The 3DSM system 500includes a local x-axis parallel to the surface 582 at each of points L,M, and N, a local z-axis normal to the surface 582 at each of points L,M, and N, and a local y-axis (not shown) orthogonal to each of x-axisand z-axis at each of points L, M, and N. The engineered illuminationsystem 530 is configured to rotate each of the ray bundles to differentrotational positions at different sample times during operation.

As mentioned above, the 3DSM system of various implementations includesan engineered illumination system configured to generate engineeredillumination and manipulate the engineered illumination. The engineeredillumination system includes one or more illumination devices, one ormore optical elements, and a motion control device. Each illuminationdevice includes one or more light sources. Various forms of lightsources can be used, for example, a light emitting diode (LED), a laser,a tungsten, and/or halogen. In various implementations, each lightsource is a point light source. In another implementation, anillumination source is in the form of a one-dimensional ortwo-dimensional array of light sources such as, for example, a twodimensional LED matrix.

According to various implementations, the engineered illumination systemcomprises one or more optical elements that generate the ray bundle ofillumination beams and propagate the illumination beams to the surfaceof the sample. Some examples of optical elements include a diffuser, acollimator, an optical fiber, optical fiber bundle, a light guide, amirror, a beam splitter, a filter such as a bandpass colored filter or apolarization filter, and other suitable optics. In certainimplementations, the one or more optical elements include at least adiffuser, a beam splitter, and a collimator.

According to various implementations, the engineered illumination systemalso includes a motion control device that is coupled to one or morecomponents of the 3DSM system. The motion control device is configuredto manipulate (rotate and/or translate) these components in order toprovide relative rotation of one or more ray bundles with respect to thesample. In addition or alternatively, the motion control device isconfigured to manipulate these components to provide engineeredillumination to different portions of a large area sample.

In embodiments having a motion control device configured to manipulatesystem components for relative rotation of one or more ray bundles, themotion control device translates and/or rotates these components todifferent positions so that each ray bundle of multiple illuminationbeams is rotated to N different rotational positions in two orthogonaldirections. At each incident angle, the motion control device holds theposition of these system components while a camera takes intensitymeasurements of light from the illuminated sample. The motion controldevice includes one or more suitable components for controlling motionsuch as, for example, actuators, motors, servos, etc. The number ofrotations in two orthogonal directions can be of different values. Insome aspects, the number of rotations is in the range of 100 to 500rotations. In one aspect, the number of rotations is greater than 100rotations. In one aspect, the number of rotations is 400 with 20rotations along one direction and 20 rotations along a second directionorthogonal to the first direction. In another aspect, the number ofrotations is 100 with 10 rotations along one direction and 10 rotationsalong a second direction orthogonal to the first direction. In anotheraspect, the number of rotations is 225 with 15 rotations along onedirection and 15 rotations along a second direction orthogonal to thefirst direction.

In certain cases, the motion control device rotates one or more raybundles by manipulating (translating and/or rotating) one or moreoptical elements of a 3DSM system. For example, the 3DSM system may havea mirror that is configured to receive illumination beams of the raybundle. In this case, the mirror can be rotated using the motion controldevice to change the rotational position of the beams. In addition oralternatively, the motion control device may manipulate the camera tocapture reflected light from the sample at different viewing directions.In one aspect, the motion control device may manipulate the beamsplitter to rotate the ray bundles. In another aspect, the motioncontrol device may also manipulate the illumination device(s) andoptical element(s) together to rotate the ray bundles.

In some embodiments, the motion control device is configured tomanipulate one or more system components to acquire intensity images fora portion of a sample (e.g., a large area sample) and then manipulatesthe one or more system components so that the sensor(s) of the cameracan acquire intensity images for a different portion of the sample untilintensity images over the entire large area sample are acquired. Forexample, the motion control device can translate the sample to differentlocations incrementally. In another example, the motion control devicecan translate the illumination device(s), optical element(s), and/or thecamera relative to the sample. In one implementation, the motion controldevice includes an x-y stage. For example, the sample platform may belocated on the x-y stage.

According to various implementations, a 3DSM system includes one or morecameras. Each camera has at least one imaging lens (e.g., a telecentriclens, a telephoto lens, or any lens with fixed or variablemagnification) and one or more sensors configured to take intensitymeasurements (also referred to herein as “intensity images,” as“images,” and as “intensity distributions”) during a data acquisitionphase of a 3DSM method. In one aspect, each sensor is an image sensor.Some examples of suitable sensors are CMOS sensors, a charge-coupleddevice (CCD), and other similar devices. Some examples of suitableimaging lens are telecentric lens, telephoto lens, lens with fixed orvariable magnification, and other similar devices. Acommercially-available example of a suitable image sensor is theDCC3260C manufactured by Thorlabs. In one aspect, a 3DSM system includesone camera. In another aspect, the 3DSM system includes multiple camerasthat can take images from multiple different views respectively.

According to various implementations, a 3DSM system includes one or moreprocessors (e.g., microprocessors) for executing instructions stored inmemory for implementing functions of the 3DSM system includingoperations of a 3DSM method. For example, the one or more processors maysend control signals to the camera to control exposures to acquireintensity images during a data acquisition phase. The one or moreprocessors may also perform operations of the 3DSM method to determine3D surface microgeometry and reflectance properties of a sample based onthe intensity images. In some cases, the instructions are stored to aninternal memory device of the controller. The internal memory device caninclude a non-volatile memory array for storing processor-executablecode (also referred to herein as “instructions”) that is retrieved bythe processor to perform various functions or operations describedherein for carrying out various logic or other operations on theintensity measurements. The internal memory device also can store rawand/or processed intensity measurement data. In some implementations,the internal memory device or a separate memory device can additionallyor alternatively include a volatile memory array for temporarily storingcode to be executed as well as measurements to be processed, stored, ordisplayed. In some implementations, the controller itself can includevolatile and in some instances also non-volatile memory.

According to various implementations, a 3DSM system includes acommunication interface in electrical communication with the controllerand a display in communication with the communication interface. Thecontroller is configured or configurable to output raw intensitymeasurements or processed data over the communication interface fordisplay on the display. In addition or alternatively, the controller canbe output raw intensity measurements or processed data over acommunication interface to an external computing system. Indeed, in someimplementations, one or more of the functions of the 3DSM system may beperformed by such an external computing system. In some implementations,the controller can be configured to store raw data as well as processeddata over a communication interface to an external memory device.

III. Examples of 3DSM Systems and Methods

In various implementations, the 3DSM system performs a 3DSM methodgenerally comprising a data acquisition phase and an analysis phase.Optionally, the 3DSM method also includes one or more of a displayphase, a calibration phase, and a phase for calculating pre-determinedsensor responses. During the data acquisition phase, one or more raybundles are rotated to N different illumination directions and at eachillumination direction, n intensity images are captured at differentexposure settings by the camera. During the analysis phase, the surfacenormal at each surface pixel is determined. To determine the surfacenormal, the 3DSM method generates a measured sensor response from thecaptured intensity images for each surface pixel, matches the measuredresponse to one of a group of pre-determined sensor responses fordifferent surface normals, and constructs a map of the surface normalsover the surface of the sample (also referred to herein as the“object”). Optionally, during the analysis phase, the 3DSM method mayalso determine properties of the surface based on the map of the surfacenormals. Because the illumination is engineered with a knowndistribution, a sensor response can be pre-calculated by taking aconvolution of the engineered beam distribution function with all thepossible surface normal directions at different illumination directions.The surface normal can then be determined based on the best matchingbetween captured sensor response and the pre-determined sensor response.A depth map can then be estimated from the surface normals. Reflectanceproperties, such as diffuse albedo, specular albedo and roughness can beestimated based on a BRDF model, such as the Ward model, theCook-Torrance model and the Blinn-Phong Model.

Various configurations of 3DSM systems for implementing 3DSM methods aredescribed in this section. Certain details of the 3DSM methods ofvarious embodiments are described in detail in Section V. Certainillustrations described in this section include some components of 3DSMsystems in order to discuss particular operations of 3DSM methods. Itwould be understood that only some of the components are illustrated inthese simplified examples and that the illustrated 3DSM systems includeadditional system components such as those described with respect toFIG. 2.

FIGS. 6A-6C are schematic diagrams illustrating three operations of andata acquisition phase performed by a 3DSM system, according to variousimplementations. FIG. 6D is a schematic diagram of operations in ananalysis phase of the 3DSM method performed by the 3DSM system of FIGS.6A-6C, according to various embodiments. The operations in theillustrated analysis phase use the intensity images captured during thedata acquisition phase illustrated in FIGS. 6A-6C to determine surfacenormals at each point along the surface and determine a depth map and/orreflectance properties based on the surface normals.

In FIGS. 6A-6C, components of the 3DSM system are shown to include anengineered illumination system 630 and a camera 660. The 3DSM systemalso includes one or more processor(s) 622 shown in FIG. 6D. In theillustrated operations of the data acquisition phase in FIGS. 6A-6C, theengineered illumination system 630 is shown providing the three L, M,and N ray bundles: a first ray bundle is incident on a point L of thesurface 682 of an sample 680, a second ray bundle is incident on a pointM of the surface 682 of an sample 680, and a third ray bundle isincident on a point N of the surface 682 of an sample 680.

Although three ray bundles are shown in FIGS. 6A-6C and other examplesfor simplicity, it would be understood that the engineered illuminationsystem can provide engineered illumination with an unlimited number ofray bundles, and that the incident illumination is two-dimensional andcan illuminate the entire surface or the entire area of the surfacebeing analyzed. In each ray bundle, there are an infinite number ofillumination beams, although three beams are shown for simplicity. Eachof the ray bundles is engineered illumination that can have power valuesthat follow certain distributions such as, for example, a Gaussiandistribution, a triangular distribution, a linear distribution, a randomdistribution, etc. During the data acquisition phase performed by theillustrated 3DSM system, the ray bundles are rotated to N differentillumination directions.

FIGS. 6A-6C shows the data acquisition operations at three of the Nillumination directions. As shown, the ray bundles are rotated to threedifferent illumination directions as shown sequentially in FIGS. 6A-6C.The reflected light R from the sample 680 propagates to the camera 660and multiple images are captured with different exposure settings. Ateach illumination direction of the ray bundles shown FIGS. 6A-6C, one ormore sensors of the camera are exposed twice, at different exposuresettings, to capture two intensity images. In FIG. 6A, the camera 660uses an Exposure I setting to capture “Image IA” and an Exposure IIsetting to capture “Image IIA.” In FIG. 6B, the camera 660 uses anExposure I setting to capture “Image IB” and an Exposure II setting tocapture “Image IIB.” In FIG. 1C, the camera 660 uses an Exposure Isetting to capture “Image IC” and an Exposure II setting to capture“Image IIC.” In another embodiment, additional images may be captured ateach illumination direction. Although the illustrated example shows aparticular angular range of illumination directions, the dataacquisition phase may include a larger angular sampling to capture amore diverse bidirectional reflectance distribution function (BRDF) inother examples. In one case, for example, an angular sampling of −45degree to +45 degree with 1 degree step can be used.

In FIG. 6D, the images 691 captured under N different illuminationdirections and under different exposure (Exposure I and Exposure II)settings are received by one or more processors 622 of the 3DSM systemin a signal from the camera 660 (shown in FIGS. 6A-6C). Because theillumination is engineered with a known distribution, a sensor response694 can be pre-determined by the one or more processors by taking aconvolution of the engineered beam distribution function with all thepossible surface normal directions at different illumination directions.Generating the pre-determined sensor response can be performed duringthe analysis phase or during a process before the data acquisitionphase, for example, during an optional calibration process. During theanalysis phase, the one or more processors determine the surface normalat each surface pixel based on the best matching between the capturedsensor response and the pre-determined sensor response. The one or moreprocessors can then estimate a depth map from the surface normals. Inaddition or alternatively, the one or more processors can determinereflectance properties, such as diffuse albedo, specular albedo androughness can be estimated based on a BRDF model, such as the Wardmodel, the Cook-Torrance model and the Blinn-Phong Model.

In various implementations, a 3DSM system captures multiple intensitymeasurements with different exposure settings at each illuminationdirection of the ray bundles. For example, one intensity measurement maybe taken at an exposure setting of 1 millisecond and a second intensitymeasurement may be taken at an exposure setting of 100 millisecond.Generally, capturing multiple exposures allows for a high dynamic rangeimage to be captured.

In implementations with a large area sample, during the data acquisitionphase the sample is translated to different translational positions andimages are captured for a smaller field of view at each translationalposition. For example, the 3DSM system might include an x-y translationstage used to translate the sample along x and y directions. Once theimages are captured at each of the smaller fields of view, the normalmap, depth map and reflectance property images determined for eachsmaller field of view and can be stitched together to achieve a largerfield-of-view.

In one implementation, a 3DSM system includes a first polarizationfilter that is configured so that the engineered illumination passesthrough the first polarization filter before propagating to the sample.The 3DSM system also includes a second polarization filter that isplaced in front of the camera. By changing the polarization stages ofthe second polarization filter placed in front of the camera, thediffuse and specular albedo images can be captured separately and thenfitted into a BRDF model to determine other reflectance properties

In various embodiments, the illumination direction of ray bundles ischanged rapidly. For example, the illumination direction may be changedat a rate of 100 degree per second. In addition, the sensors may takeintensity measurements at a rate of 100 frames per second.

FIG. 7 is a schematic drawing of a 3DSM system 700, according to anembodiment. The 3DSM system 700 comprises an engineered illuminationsystem having optical elements of a diffuser 736, a mirror 737, and abeam-splitter 738. The engineered illumination system further comprisescollimated illumination 735 from an illumination device (not shown). The3DSM system 700 further comprises a camera 760. During the illustrateddata acquisition phase, the diffuser 736 receives collimatedillumination 735 and the collimated illumination is transmitted throughthe diffuser 736 spreading the illumination based on the diffuser'stransmittance which typically follows a distribution such as a Gaussiandistribution. For example, the diffuser 736 may have one of thetransmission profiles shown in FIG. 3A. In FIG. 7, the diffuser 736spreads out the collimated illumination into three illumination beamsA1, A2, and A3. The illumination beams A1, A2 and A3 are with differentintensities after being transmitting through the diffuser 736. Theillumination beam A2 has the highest intensity and illumination beams A1and A3 have lower intensities. The mirror 737 receives the threeillumination beams A1, A2, and A3 and reflects these beams. The beamsplitter 738 receives the reflected beams B1, B2, and B3 from the mirror737 and reflects these beams. The reflected beams M1, M2, and M3 arepropagated to the point M at the surface 782 of the sample 780. In theillustration, a ray bundle of three illumination beams M1, M2, and M3 isshown for simplicity, it would be understood that the engineeredillumination provided to point M has an infinite number of illuminationbeams. In the illustration, only one ray bundle is shown at surfacepoint M, it would be understood that infinite number of surface pointsreceive infinite number of engineered illumination bundles. The surface782 reflects an illumination beam and the reflected beam 792 istransmitted through the beam splitter 738 and received at the camera760. The intensity distribution of the incident beams M1, M2, and M3 isthe same of illumination beams A1, A2, and A3. That is, the incidentbeams M1, M2, and M3 are engineered with a known distribution functionwhich is the same distribution function of the illumination beams A1-A3.Although three beams are shown for simplicity, it would be understoodthat the same would apply for a larger number of beams.

FIG. 8 is a sensor response captured without engineered illumination. Inthis example, the intensity image is captured with incident lightparallel to a camera's optical axis. The incident beam is not engineeredand is only collimated illumination. The sample is a chrome sphere whichhas mirror-like reflection and surface normal elevation angle changedfrom 0 to 90 degree.

FIG. 9 is a sensor response determined using the 3DSM system of FIG. 7.In this example, the intensity image is captured with engineeredillumination following a Gaussian distribution implemented with the useof a diffuser. By comparing the intensity images of FIG. 8 and FIG. 9,it is shown that under regular collimated illumination as illustrated inFIG. 8, only a few pixels of the camera measure intensity. In this case,the limited number of pixel values would be available to estimate asurface normal. In comparison, under engineered illumination as shown inFIG. 9, the sensor response is much broader, and many more pixel valuescan be used for surface normal estimation. Technical advantages ofcertain embodiments using engineering illumination with a knowndistribution function over using collimated light may include that thespatial range of surface normals to be constructed can be significantlyincreased and/or each surface normal can be estimated with higherangular resolution.

FIG. 10A is a schematic drawing of components of a 3DSM system 800,according to an embodiment. The 3DSM system 800 comprises an engineeredillumination system having optical elements including a collimator 834,a fiber bundle 836, a combination filter 835, a mirror 837, and abeam-splitter 838. The 3DSM system 800 further comprises a camera 860.FIG. 10B shows an example layout of the combination filter 835 shown inFIG. 10A, according to one implementations. In FIG. 10B, the combinationfilter 835 is a combination of three neutral density (ND) filters withannulus shape: a first ND filter 895, a second ND filter 896, and athird ND filter 897. Although three ND filters are shown, otherimplementations may use additional filters. The first ND filter 895 inthe middle has the highest transmittance, and transmittance of thesecond ND filter 896 and the third ND filter 897 is gradually decreased.FIG. 10C is another example layout of the combination filter 835 shownin FIG. 10A, according to an implementation. In this example, the filterhas a randomly distributed transmission. FIG. 10D is another example ofthe combination filter 835 shown in FIG. 10A, according to animplementation. In this example, the filter has a linear gradientdistributed transmission.

Returning to FIG. 10A, during the illustrated data acquisition phase,illumination passes through the fiber bundle 836 to the combinationfilter 835. The illumination passing through the different partitions ofthe combination filter 835 are engineered into three beams withdifferent intensities from different points on combination filter 835.After passing through the collimator 834, the beams A1, A2, and A3 havedifferent intensities, which is similar to the beams from the diffuser736 shown in the configuration of FIG. 7. The illumination beam A2 hasthe highest intensity and illumination beams A1 and A3 have lowerintensities. The mirror 837 receives the three illumination beams A1,A2, and A3 and reflects these beams. The beam splitter 838 receives thereflected beams B1, B2, and B3 from the mirror 837 and reflects thesebeams. The reflected beams M1, M2, and M3 are propagated to the point Mat the surface 882 of the sample 880. In the illustration, a ray bundleof three illumination beams M1, M2, and M3 is shown for simplicity, itwould be understood that the engineered illumination provided to point Mhas an infinite number of illumination beams. The surface 882 reflectsan illumination beam and the reflected beam 892 is transmitted throughthe beam splitter 838 and received at the camera 860. The intensitydistribution of the incident beams M1, M2, and M3 is the same as theintensity distribution of illumination beams A1, A2, and A3.

FIG. 41 is a schematic drawing depicting a data acquisition operation ofa 3DSM system 4100, according to an embodiment. The 3DSM system 4100comprises an illumination source 4134 and an engineered illuminationsystem having optical elements including a diffuser 4136, anillumination collimator 4139, a mirror 4137, and a beam-splitter 4138.The 3DSM system 4100 further comprises a camera 4160. During theillustrated data acquisition phase, the light from an illuminationsource 4134 is passed through the diffuser 4136, which is engineeredwith certain transmission distribution, such as a Gaussian distribution.The light passed through the diffuser 4136 is then collimated by anillumination collimator 4139. The beams A1, A2, and A3 are furtherreflected by a beam splitter 4138 and are projected onto a surface 4182of a sample 4180. The incident beams L1, M2, and N3 are also engineeredwith a known intensity distribution that follows the transmissionprofile of the diffuser 4136.

A. Manipulation of Illumination Direction

In certain implementations, a 3DSM system is configured to manipulatethe illumination direction of one or more ray bundles. For example, a3DSM system may include a moving mirror to manipulate the direction ofillumination beams reflected from the mirror. FIGS. 11A-11B areschematic diagrams illustrating data acquisition operations performed bya 3DSM system 900 with a rotating (spinning) mirror 937, according to anaspect. FIG. 12 is a schematic diagram illustrating a data acquisitionoperation performed by the 3DSM system 900 of FIGS. 11A-B. In one case,the diagrams in FIGS. 11A-11B and FIG. 12 illustrate three operations ofa data acquisition phase. In another case, the diagrams in FIGS. 11A-11Billustrate two operations of operations of a data acquisition phase andthe diagram in FIG. 12 illustrates an operation of another dataacquisition phase.

The 3DSM system 900 comprises an engineered illumination system havingoptical elements including a diffuser 936, a rotating mirror 937, and abeam-splitter 938. The engineered illumination system also includescollimated illumination 935 from an illumination device (not shown). The3DSM system 900 further comprises a camera 960, for example, of acamera. The 3DSM system 900 also includes an x-axis, a y-axis, and az-axis (not shown) orthogonal to the x-axis and the y-axis. The 3DSMsystem 900 also includes a motion control device 932 coupled to therotating mirror 937. The motion control device 932 is configured torotate the rotating mirror 937 about the x-axis and the z-axis (twoorthogonal rotational directions). The illustrations show the rotatingmirror 937 being rotated about the z-axis between a first position(dotted line) and the current second position (solid line).

In FIGS. 11A and 11B, the diffuser 936 receives collimated illumination935 and the collimated illumination is transmitted through the diffuser936 spreading the illumination based on the diffuser's transmittancedistribution. The mirror 937 receives the illumination beam A2 andreflects it. The beam splitter 938 receives the reflected beam B2 fromthe mirror 937 and reflects it. The reflected beam M2 is propagated to apoint at the surface 982 of the sample 980. A surface 982 of a sample980 reflects the illumination beam and the reflected beam 992, 993 istransmitted through the beam splitter 938 and the reflected transmittedbeam 992, 993 is received at the camera 960.

In FIG. 12, the diffuser 936 receives collimated illumination 935 andthe collimated illumination is transmitted through the diffuser 936spreading the illumination based on the diffuser's transmittancedistribution. The rotating mirror 937 receives the illumination beam A2and reflects it. The surface 982 of the sample 980 reflects theillumination beam B2 and the reflected beam 994 is transmitted throughthe beam splitter 938 and the reflected transmitted beam 994 is receivedat the camera 960. The reflected transmitted beam 994 is received at thecamera 960. In FIG. 12, when the rotating mirror 937 is oriented with alarge oblique angle, the illumination beam B2 is directly reflected fromthe rotating mirror 937 onto the surface 982 without bouncing from thebeam splitter 938. This illustrated configuration allows for a muchlarger beam incident angle.

In FIGS. 11A-11B, the direction of the incident illumination beam M2 canbe changed by rapidly spinning the rotating mirror 937 with the motioncontrol device 932, such as a rotation stage, a gimbal mount or agoniometer. In FIG. 12, the direction of the incident illumination beamB2 can be changed by rapidly spinning the rotating mirror 937 with themotion control device 932. Although a single incident beam is shown inFIGS. 11A-B and FIG. 12, it would be understood that the same systemconfiguration can be used to direct incident beam bundles ofillumination beams across the full field-of-view such as shown in FIGS.6A-6C.

A 3DSM system configured to manipulate the engineered illuminationdirection of one or more ray bundles with a spinning mirror may provideone or more technical advantages. First, one technical advantage is thata large number of angular samplings of the BRDF can be captured aroundthe predominant surface normal, and very fine sampling step may beachieved by adjusting the mirror spinning angle. Such an approach mayallow for unbiased estimation of surface normals for specular surfacesthat are relatively smoother. Second, one technical advantage is thatthis implementations may avoid shadow effects, which are common forphotometric stereo techniques. Third, another technical advantage isthat by using engineered illumination the range of surface normals thatcan be estimated is significantly extended. If not using engineeredillumination, the range of surface normals to be estimated is solelydetermined by the scanning range of the incident light. But withengineered illumination the range of surface normals that can bemeasured is increased, and the enhancement is proportional to thebandwidth of the engineered illumination distribution. Fourth, anothertechnical advantage is that the camera and illumination do not need tobe placed at far distance in order to fulfill the orthographicassumptions. Therefore, the whole system can be built in a compact formfactor and is able to capture data at a closer distance to achievebetter spatial and depth resolution and accuracy.

For samples having surfaces with large slopes, additional illuminationsources can be included at an oblique angle to improve estimation. FIG.13 illustrates an example of the placement of additional illuminationsources. Although only two illumination sources are shown in theillustrated example, additional sources may be used in otherembodiments.

FIG. 13 is a schematic diagram illustrating a 3DSM system 1000 with arotating (spinning) mirror 1037, according to an aspect. The 3DSM system1000 comprises an engineered illumination system having optical elementsincluding a diffuser 1036, a rotating mirror 1037, and a beam-splitter1038. The engineered illumination system also includes threeillumination devices: a first illumination device 1010 configured toprovide collimated illumination 1035, a second illumination device 1011,and a third light source 1012. The 3DSM system 1000 further comprises acamera 1060. The 3DSM system 1000 also includes an x-axis, a y-axis, anda z-axis (not shown) orthogonal to the x-axis and the y-axis. The 3DSMsystem 1000 also includes a motion control device 1032 coupled to therotating mirror 1037. The motion control device 1032 is configured torotate the rotating mirror 1037 about the x-axis and the z-axis (twoorthogonal rotational directions). The illustrations show the rotatingmirror 1037 being rotated about the z-axis between a first position(dotted line) and the current second position (solid line). The diffuser1036 receives collimated illumination 1035 and the collimatedillumination is transmitted through the diffuser 1036 spreading theillumination based on the diffuser's transmittance distribution.Although a single illumination beam from the diffuser 1036 is shown forsimplicity, it would be understood that multiple beams are passed to therotating mirror 1037. The rotating mirror 1037 receives the illuminationbeam A2 and reflects it. The beam splitter 1038 receives the reflectedbeam B2 from the mirror 1037 and reflects it. The reflected beam M2 ispropagated to a point at the surface 1082 of the sample 1080. A surface1082 of a sample 1080 reflects the incident beam. Illumination reflectedfrom the surface 1082 is transmitted through the beam splitter 1038. Thebeam 1093 propagated from the beam splitter 1038 is received at thecamera 1060.

FIGS. 14A-14B are schematic diagrams illustrating data acquisitionoperations performed by a 3DSM system 1100 configured for illuminationdirection manipulation, according to an embodiment. The 3DSM system 1100comprises an engineered illumination system having optical elementsincluding a diffuser 1136, a collimator 1134 with an optical axis 1133,an illumination array 1135, a mirror 1137, and a beam-splitter 1138. The3DSM system 1100 further comprises a camera 1160. As shown, theillumination array 1135 is placed behind the collimator 1134.

Different types of illumination can be used in the illumination array1135, such as LEDs and optical fibers. In the illustrated example, themiddle illumination element (e.g., LED) is located at the focal point ofthe collimator 1134. In other examples, other positions may be used.Each circle of the illumination array 1135 represents an individualillumination element. A solid circle denotes an illumination elementthat is turned on, and empty circle represent an illumination that isturned off. By turning each individual illumination element on and off,the direction of incident beam M2 to the surface 1182 of the sample 1180can be manipulated. In FIGS. 14A-14B, the illumination array 1135includes five illumination elements 1191, 1192, 1193, 1194, and 1195. Inother example, more or fewer illumination elements can be used. In FIG.14A, the middle illumination element 1193 of the illumination array 1135is illuminated. In FIG. 14B, an illumination element 1191 of theillumination array 1135 is illuminated. The collimator 1134 receives theillumination from the activated element.

In the operations illustrated in FIGS. 14A-14B, the diffuser 1136receives collimated illumination from the collimator 1134. Illuminationtransmitted through the diffuser 1136 is spread based on the diffuser'stransmittance distribution. The mirror 1137 receives the illuminationbeams and reflects them. For simplicity, a single beam A2 is shownreflected by mirror 1137. The beam splitter 1138 receives the reflectedbeam B2 from the mirror 1137 and reflects it. The reflected beam M2 ispropagated to a point at the surface 1182 of the sample 1180. A surface1182 of a sample 1180 reflects the illumination beam and the reflectedbeam 1198 is transmitted through the beam splitter 1138 and thereflected transmitted beam 1198 is received at camera 1160. Duringoperation, the 3DSM system 1100 can manipulate the direction of theincident beam M2 by turning individual illumination elements on and off.

B. Calibration Process

In certain implementations, the 3DSM system generates engineeredillumination of ray bundles that are aligned to the same direction. InFIG. 6A-6C, for example, the center illumination beams L2, M2, and N2 ofthe three ray bundles are parallel to each other. In otherimplementations, however, the illumination beams may diverge or convergedue to imperfect beam collimation. In one implementation, the 3DSMincludes a calibration process that can be used to estimate the offsetof the incident beams from the ideal condition discussed above.

The directions of incident beams in a 3DSM system (e.g., incident beamsL2, M2, and N2 in FIGS. 6A-6C) can be controlled by illumination control(e.g., using the spinning mirror 937 in FIG. 11A) or based on theknowledge of an illumination array (e.g., illumination array 1135 inFIG. 14A). To account for misalignments in a 3DSM system, a calibrationprocess can be used to determine offsets and the offsets can beaccounted for in the analysis phase.

Below are examples of calibration processes that use various elements asreference objects. The illustrated examples can be used in any of thesystem configurations described herein.

1. Mirror

In other implementations, a system calibration process uses a mirror.One example of a system calibration process using a mirror 1480 is shownin FIGS. 15A-15C, in accordance with an embodiment. FIGS. 15A-15Cillustrate three data acquisition operations during the calibrationprocess implemented by a 3DSM system, according to an embodiment. Duringthe calibration process, the mirror 1480 is placed as the sample beinganalyzed by the 3DSM system, and directions of the incident beams arechanged. The 3DSM system includes a camera 1460. During the calibrationprocess, the illumination reflected from mirror 1480 is received at thecamera 1460 and intensity images are captured at all the differentdirections. FIGS. 15A-15C illustrate three directions of the incidentbeams. The 3DSM system has one or more processors (e.g., processor(s) 22in FIG. 2) that can calibrate the deviation of the incident beams basedon the reflection from the mirror 1480.

FIGS. 16A-16B are illustrations showing the calibrated offset ofincident beams resulting from the calibration process using a mirror,according to an embodiment. FIG. 16A shows an illustration of thecalibrated offset of incident beams in terms of the zenith angle at eachpixel. FIG. 16B shows an illustration of the calibrated offset ofincident beams in terms of the azimuth angle at each pixel. Ideally thevalues for the zenith angle and azimuth angle should be the same at eachpixel if a perfect collimation is achieved. But in practice, the valuesare different at each pixels as shown in FIGS. 16A and 16B. In oneimplementation, the 3DSM method uses the results of the calibrationprocess to account for the offset in the analysis phase to compensatefor any misalignments.

FIGS. 17A and 17B are illustrations showing system alignment using thecalibration process using a mirror as a reference object, according toan embodiment. In FIG. 6A, the central beams L2, M2, and N2 in each raybundle are perpendicular to the object surface and the reflected beam Ris parallel to these central beams. In certain implementations, thisparallel arrangement is preferred. In order to have this parallelalignment, alignment of different system components in the illuminationpath and imaging path is required such as can be accomplished using acalibration process described herein. In one implementation, thecalibration process uses a mirror in a similar method as shown in FIGS.15A-15C for active alignment of the optics (also referred to herein asoptical elements), the camera, and the object plane, for example, bymanipulating a sample platform. FIGS. 17A and 17B are examples ofcaptured images of a reference mirror surface using the 3DSM system withthe configuration shown in FIGS. 11A-11B. FIG. 17A is a captured imagethat is uniform, which indicates alignment of different opticalcomponents. FIG. 17B is a captured image that is not uniform, whichindicates poor alignment of one or more optical components. Thecalibration method using a mirror works because the mirror is flat andcan only reflect light in one direction, and thus any misalignment canresult in partially reflected beams from the mirror.

2. Chrome Sphere(s)

In other implementations, a system calibration process uses one or morechrome spheres. An example of system calibration process using a chromesphere array 1580 is shown in FIGS. 18A-18C, in accordance with anembodiment. Different number of chrome spheres can be used in the chromesphere array 1580.

FIGS. 18A-18C illustrate three data acquisition operations during thecalibration process implemented by a 3DSM system 1500, according to anembodiment. During the calibration process, the chrome sphere array 1580is placed as the object, and directions of the incident beams arechanged. The 3DSM system 1500 includes a camera 1560. During thecalibration process, the illumination reflected from the chrome spherearray 1580 is received at the camera 1560 and intensity images arecaptured at all the different directions. Because each sphere has knowngeometry, when the incident beams change direction, different parts ofthe sphere are highlighted. The 3DSM system has one or more processors(e.g., processor(s) 22 in FIG. 2) that can calibrate the incident beamdirections based on the reflection from the chrome sphere array 1580.

FIGS. 19A and 19B are captured images of a single chrome sphere of achrome sphere array when the incident beams are changed to two differentdirections, and different parts of sphere show highlight, according toan implementation. Based on the position of the highlight, the directionof the incident beam can be estimated.

FIGS. 20A and 20B are illustrations of examples of calibrated zenithangle and azimuth angle for 169 different incident beam directions usinga calibration process of the 3DSM system 1500 illustrated in FIGS.17A-17C, according to an implementation.

3. Mirror and Chrome Sphere(s)

In other implementations, a system calibration process uses a mirror andone or more chrome spheres. FIGS. 21A-21C is a schematic diagram of a3DSM system 2000 implementing a system calibration process using acombination of a mirror 2080 and a single chrome sphere 2081, inaccordance with an embodiment. During the calibration process, themirror 2080 and the chrome sphere 2081 are placed as the object beingimaged, and the directions of the incident beams are changed. The 3DSMsystem 2000 includes a camera 2060. During the calibration process, theillumination reflected from the chrome sphere 2081 and from the mirror2080 is received at the camera 2060 and intensity images are captured atall the different directions. The offset of beams from ideal parallelcondition is calibrated using the method depicted in FIGS. 15A-15C andFIGS. 16A-16B, and the incident beam directions are calibrated using themethod depicted in FIGS. 18A-18C, FIGS. 19A-19B, and FIGS. 20A-20B. Byusing combined mirror and chrome sphere these two calibration processescan be performed simultaneously.

C. Pre-Determined Sensor Responses (Also Referred to Herein asPre-Computed Sensor Responses)

Because the engineered illumination has a known intensity distribution,sensor response can be pre-determined by taking the convolution of theengineered beam distribution function with all the possible surfacenormal directions at different illumination directions. In variousimplementations, a 3DSM method includes operations during a dataacquisition phase that acquire intensity images and operations during ananalysis phase that use the intensity images to determine a measuredsensor response for each surface point and matches the measured sensorresponse to a pre-determined sensor response to determine a surfacenormal at each surface point that corresponds to a pixel of the imagedata of the surface. A surface point is sometimes referred to herein asa “pixel” or a “surface pixel.”

FIG. 22 depicts a series of pre-determined sensor responses of a singlesensor pixel, according to an embodiment. Each sensor response is in theform of a three-dimensional plot. In each sensor response plot, thex-axis and y-axis represents the incident beam direction changed in thex and y directions respectively, and the z-axis is normalized intensityof the sensor response. Each plot is the pre-determined sensor responsewith a different surface normal vector. The illustrated series of ppre-determined sensor responses includes a first sensor response 1902, asecond sensor response 1904, . . . , a (p−1)^(th) sensor response 1906,a p^(th) sensor response 1908.

FIG. 23 is a schematic representation of the operation of matching ameasured sensor response for a single sensor pixel corresponding to asurface pixel 1911 to one of a series of pre-determined sensorresponses, according to an embodiment. In FIG. 23, the captured response1912 is calculated for a single pixel 1911 when the illumination changedin both x and y directions, and a best matched pre-determined sensorresponse 1914 can be found based on different metrics, such ascorrelation and minimum difference. The surface normal vector of thematched pre-determined sensor response is then assigned to that pixel.The same sensor response matching process is repeated for all the pixelsin the captured images.

D. Examples of Microgeometry and Reflectance Properties Measurements,and Related Renderings

FIG. 24A is a photograph of a leather sample. The area 2301 of theleather sample (denoted by the rectangle in the photograph) was analyzedby a 3DSM system of an implementation to measure microgeometry in theform of a depth map and reflectance properties. According to one aspect,the area 2301 may be the portion of the sample within the field-of-viewof the camera.

FIG. 24B is an illustration of a measured depth map of the area 2301 ofthe leather sample shown in FIG. 24A as analyzed by the 3DSM system.FIG. 24C is a surface normal map of the portion 2301 of the leathersample as analyzed by the 3DSM system. FIG. 24D is a measured diffusemap of the portion 2301 of the leather sample as analyzed by the 3DSMsystem. FIG. 24E is a measured specular map of the area 2301 of theleather sample as determined by the 3DSM system. FIG. 24F is a measuredsurface roughness map of the area 2301 of the leather sample asdetermined by the 3DSM system.

FIG. 25A is a photograph of a coin. FIG. 25B is a 3D rendering of thecoin shown in FIG. 25A that is generated by a 3DSM system using ameasured depth map, according to an embodiment. FIG. 25C is graph havinga measured depth curve measured along the black line of the coin shownin FIG. 25A as determined by the 3DSM system. The depth is measured inmicron.

FIGS. 26A-26D are physically-based renderings based on measuredmicrogeometry and reflectance properties of samples measured using an3DSM system according to an implementation. FIG. 26A is a rendering of awool sample. FIG. 26B is a rendering of green leather sample. FIG. 26Cis a rendering of a rock sample. FIG. 26D is a rendering of a laminatedwood sample.

E. Wavelength and Polarization Control

In certain implementations, a 3DSM system controls wavelength and/orpolarization of the engineered illumination, the light from the samplesurface (in optical path between the camera and the sample), or both theengineered illumination and the light from the sample surface. In thesecases, the 3DSM includes one or more wavelength and/or polarizationcontrol devices. To control the wavelength of the engineeredillumination, the reflected light, or the light at another point in theoptical path to the camera, a control device such as filter wheel loadedwith different spectral filters or a liquid crystal tunable filter canbe used. Alternatively, different light sources with differentwavelength can also be used to control the wavelength of the engineeredillumination. To control the polarization, a polarizer rotated atdifferent positions or a filter wheel loaded with differ polarizers canbe used.

FIGS. 27A-27C are schematic diagrams depicting three operations of adata acquisition phase implemented by a 3DSM system 2700 configured tocontrol wavelength and polarization of the engineered illumination, inaccordance with an embodiment. The 3DSM system 2700 comprises anengineered illumination system 2730, a camera 2760, and awavelength/polarization control device 2739 in communication with theengineered illumination to change the wavelength and/or polarizationstates of the engineered illumination. In one case, thewavelength/polarization control device 2739 is a component of theengineered illumination system 2730. The engineered illumination'swavelength and polarization can be controlled alone or be controlledtogether, and images are captured under different illuminationdirections and under different exposures in a way that is similar to theway shown in FIGS. 6A-6C. In the illustrated operations, the engineeredillumination system 2730 is shown providing three ray bundles: a firstray bundle is incident on a point L of the surface 2782 of an sample2780, a second ray bundle is incident on a point M of the surface 2782of the sample 2780, and a third ray bundle is incident on a point N ofthe surface 2782 of the sample 2780.

FIGS. 27A-27C show the data acquisition operations at three of the Nillumination directions of the ray bundles. The reflected light R fromthe sample 2780 propagates to the camera 2760 and multiple images arecaptured at different wavelength/polarization states at each of the twoexposure settings “Exposure A′” and “Exposure B.” Although threeillumination directions and two exposures at different settings areshown for simplicity, it would be understood that data acquisition caninclude additional illumination directions and/or additional exposures.The captured images are then communicated in a signal to the one or moreprocessors to analyze the images for different wavelength andpolarization states. During an analysis phase, the one or moreprocessors use the captured images to construct a map of surfacenormals, a depth map and maps of reflectance properties, all measured atdifferent wavelength and/or polarization states.

FIGS. 28A-28C are schematic diagrams illustrating three operations of adata acquisition phase that control wavelength and polarization of thereflected light 2898 from the surface 2882 of the sample 2880 using a3DSM system 2800, in accordance with an embodiment. The 3DSM system 2800comprises an engineered illumination system 2830, a camera 2860, and awavelength/polarization control device 2839 in communication with thereflected light 2898 to change the wavelength and/or polarizationstates. In one case, the wavelength/polarization control device 2839 isa component of the engineered illumination system 2830. The wavelengthand polarization of the reflected light 2898 can be controlled alone orbe controlled together, and images are captured under differentillumination directions and under different exposures in a way that issimilar to the way shown in FIGS. 6A-6C. In the illustrated operations,the engineered illumination system 2830 is shown providing three raybundles: a first ray bundle is incident on a point L of the surface 2882of a sample 2880, a second ray bundle is incident on a point M of thesurface 2882 of the sample 2880, and a third ray bundle is incident on apoint N of the surface 2882 of the sample 2880. FIGS. 28A-28C shows thedata acquisition operations at three of the N illumination directions ofthe ray bundles. The reflected light R from the sample 2880 propagatesto the camera 2860 and multiple images are captured at differentwavelength/polarization states at each of the two exposure settings“Exposure A′” and “Exposure B.” Although three illumination directionsand two exposures at different settings are shown for simplicity, itwould be understood that data acquisition can include additionalillumination directions and/or additional exposures. The captured imagesare then communicated in a signal to the one or more processors toanalyze the images for different wavelength and polarization states.During operations of an analysis phase of the 3DSM method, the one ormore processors use the captured images to construct a map of surfacenormal, a depth map and maps of reflectance properties, all measured atdifferent wavelength and/or polarization states.

FIGS. 29A-29C are schematic diagrams illustrating three operations of adata acquisition phase that control wavelength and polarization of boththe engineered illumination and the reflected light 2998 from thesurface 2982 of the sample 2980 using a 3DSM system 2900, in accordancewith an embodiment. The 3DSM system 2900 comprises an engineeredillumination system 2930, a camera 2960, a first wavelength/polarizationcontrol device 2938 in communication with the engineered illumination tochange the wavelength and/or polarization states of the engineeredillumination, and a second wavelength/polarization control device 2939in communication with the reflected light 2998 to change the wavelengthand/or polarization states of the reflected light 2998. In one case, oneor both of the first and second wavelength/polarization control devices2938, 2939 are components of the engineered illumination system 2930.The engineered illumination's wavelength and polarization can becontrolled alone or be controlled together, and the wavelength andpolarization of the reflected light 2998 can also be controlled alone orbe controlled together.

Images are captured under different illumination directions and underdifferent exposures in a way that is similar to the way shown in FIGS.6A-6C. In the illustrated operations, the engineered illumination system2930 is shown providing three ray bundles: a first ray bundle isincident on a point L of the surface 2982 of an sample 2980, a secondray bundle is incident on a point M of the surface 2982 of the sample2980, and a third ray bundle is incident on a point N of the surface2982 of the sample 2980. FIGS. 29A-29C show the data acquisitionoperations at three of the N illumination directions of the ray bundles.The reflected light R from the sample 2980 propagates to the camera 2960and multiple images are captured at different wavelength/polarizationstates at each of the two exposure settings “Exposure A′” and “ExposureB.” Although three illumination directions and two exposures atdifferent settings are shown for simplicity, it would be understood thatdata acquisition can include additional illumination directions and/oradditional exposures. The captured images are then communicated in asignal to the one or more processors to analyze the images for differentwavelength and polarization states. During an analysis phase, the one ormore processors use the captured images to construct a map of surfacenormals, a depth map and maps of reflectance properties, all measured atdifferent wavelength and/or polarization states.

FIG. 30 is a schematic diagram illustrating an operation of a dataacquisition phase of a 3DSM system 3000 that controls wavelength andpolarization of both the engineered illumination and the reflected light3098 from the surface 3082 of the sample 3080, in accordance with anembodiment. The 3DSM system 3000 comprises an engineered illuminationsystem having an illumination source 3034 and optical elements includinga first wavelength/polarization control device 3031, a collimator 3033,a diffuser 3036, a rotating mirror 3037, and a beam-splitter 3038. The3DSM system 3000 further comprises a second wavelength/polarizationcontrol device 3039 and the camera 3060. The 3DSM system 3000 alsoincludes an x-axis, a y-axis, and a z-axis (not shown) orthogonal to thex-axis and the y-axis. The 3DSM system 3000 also includes a motioncontrol device 3032 coupled to the rotating mirror 3037. The motioncontrol device 3032 is configured to rotate the rotating mirror 3037about the x-axis and the z-axis (two orthogonal rotational directions).The first wavelength/polarization control device 3031 receivesillumination propagated from the illumination source 3034 and thecollimator 3033 receives light propagated from the firstwavelength/polarization control device 3031. During operation, thediffuser 3036 receives collimated illumination 3035 from the collimator3033 and the diffuser 3036 spreads the illumination based on thediffuser's transmittance distribution. For simplicity, threeillumination beams A1, A2, A3 are shown. The rotating mirror 3037receives the illumination beams and reflects them. The beam splitter3038 receives the reflected beams B1, B2, B3, from the mirror 3037 andreflects them. The surface 3082 of the sample 3080 receives the incidentbeams L1, M2, and N3 at the points L, M, and N respectively. Lightreflected from the surface 3082 is transmitted through the beam splitter3038. The second wavelength/polarization control device 3039 receiveslight transmitted through the beam splitter 3038. Light propagated bythe second wavelength/polarization control device 3039 is received atthe camera 3060 and multiple images are captured at differentwavelength/polarization states. During operation, the firstwavelength/polarization control device 3031 controls the wavelengthand/or polarization states of the light propagated to the collimator3033 and/or the second wavelength/polarization control device 3039controls the wavelength and/or polarization states of the lightreflected from the sample 3080 and transmitted through the beam splitter3038. During operation, light is propagated to the camera 3060 andmultiple images are captured at different wavelength/polarization statesat each of the multiple exposure settings. The captured images arecommunicated in a signal to the one or more processors to analyze theimages for different wavelength and polarization states.

In FIG. 30, the first wavelength/polarization control device 3031implements wavelength and polarization control of the engineeredillumination and the second wavelength/polarization control device 3039implements wavelength and polarization control of the reflected lightfrom the sample surface. To control the wavelength a device such asfilter wheel loaded with different spectral filters or liquid crystaltunable filter can be used. To control the polarization a polarizerrotated at different positions or a filter wheel loaded with differentpolarizers can be used. Different light sources with differentwavelength can also be used to control the wavelength of engineeredillumination.

F. Controlled Motion of Sample Surface

In certain implementations, a 3DSM system is configured to control themotion of the sample surface. For example, a 3DSM system may include amotion control device coupled to the sample platform to control thetranslation, tilt, and rotation of the sample surface.

FIGS. 31A-31C are schematic diagrams illustrating three operations of adata acquisition operations performed by a 3DSM system 3100 including amotion control device 3132 coupled to a sample platform 3140 formanipulating the position of the sample surface, according to animplementation. The 3DSM system 3100 comprises a sample platform 3140,an engineered illumination system 3130, a motion control device 3132coupled to a sample platform 3140. In the illustrated operations of thedata acquisition phase in FIGS. 31A-31C, the incident illuminationincludes three ray bundles: a first ray bundle is incident on a point Lof the surface 3182 of an sample 3180, a second ray bundle is incidenton a point M of the surface 3182 of an sample 3180, and a third raybundle is incident on a point N of the surface 3182 of an sample 3180.Although three ray bundles with three beams are shown for simplicity, itwould be understood that larger numbers of beams and/or larger number ofray bundles may be used. The motion control device 3132 is configured tocontrol the position of the sample including one or more of translation,tilt, and rotation applied to the sample surface. At each controlledposition of the sample 3180, the camera captures images under differentillumination directions and under different exposures settings similarto the images captured in FIGS. 6A-6C. Although three illuminationpositions are shown, it would be understood that additional illuminationpositions may be implemented. During operation, the image data of thecaptured images is communicated in a signal to the one or moreprocessors and the processors use the image data to construct a map ofsurface normals, a depth map and maps of reflectance properties at eachcontrolled position.

In implementations where the surface of a sample is larger than thecamera's field-of-view, the sample surface can be translated alongdifferent directions in the sample plane, and images captured atdifferent translation positions. The images captured at each controlledposition can be used to measure the map of surface normals, depth map,and reflectance properties at each translation position and then thisdata can be stitched together.

Height Mapping and Anisotropic Properties

In some implementations, a 3DSM system further comprises a stage that isconfigured to adjust the distance between the surface of the sample andthe camera to bring the surface into focus. In one implementation, thedistance between the surface and the camera can be adjusted graduallyand images captured at each distance. Images captured at the differentdistances can be used to determine properties and can be combinedtogether to extend the depth of field of the camera, and can also beprocessed to estimate a height map based on the depth from defocusmethod. This height map can be combined with the depth map measuredusing the engineered illumination.

In some scenarios, a surface may present anisotropic reflection. In thiscase, it can be useful to measure properties at different rotationalpositions. In one implementation, the 3DSM system further comprises arotational stage that can be used to rotate the sample surface, and the3DSM system measures surface normal, depth, and reflectance propertiesat different rotation positions. The surface can be tilted to change thegeometric relationship among illumination, surface normal, and cameraviewing direction.

In some cases, images captured at these additional height and/orrotational positions can be fitted into a BRDF model together with theimages captured at different illumination directions, and can provideimproved reflectance properties measurement.

G. Controlled Motion of Engineered Illumination and Camera

In certain implementations, a 3DSM system is configured to control themotion of the engineered illumination and the camera together in unison.For example, a 3DSM system may include a motion control deviceindividually coupled each of the engineered illumination system and thecamera individually or may be coupled with both.

FIGS. 32A-32C are schematic diagrams illustrating three operations of adata acquisition phase performed by a 3DSM system 3200 configured tocontrol the motion of the engineered illumination and the camera,according to an embodiment. Engineered illumination and camera can becontrolled together or be controlled individually by applying differentmotion controls such as translation, tilt, and rotation. At eachcontrolled position, images are captured under different illuminationdirections and under different exposures in a way that is similar to theoperations described with respect to FIGS. 6A-6C. The 3DSM system 3200comprises an engineered illumination system 3230, a camera 3260, and amotion control device 3232 coupled to both the engineered illuminationsystem 3230 and the camera 3260. In the illustrated operations of thedata acquisition phase in FIGS. 32A-32C, the incident illuminationincludes three ray bundles: a first ray bundle is incident on a point Lof the surface 3282 of an sample 3280, a second ray bundle is incidenton a point M of the surface 3282, and a third ray bundle is incident ona point N of the surface 3282. Although three ray bundles with threebeams are shown for simplicity, it would be understood that largernumbers of beams and/or larger number of ray bundles may be used. Themotion control device 3232 is configured to control one or more oftranslation, tilt, and rotation applied to the engineered illuminationsystem 3230 and the camera 3260. At each controlled position, the camera3260 captures images under different illumination directions and underdifferent exposures settings similar to the images captured in FIGS.6A-6C. Although three positions are shown, it would be understood thatadditional positions may be implemented. During operation, the imagedata of the captured images is communicated in a signal to the one ormore processors and the processors use the image data to construct a mapof surface normals, a depth map, and maps of reflectance properties ateach controlled position.

In implementations where the surface of a sample is larger than thecamera's field-of-view, the engineered illumination and camera can betranslated together or be translated individually along differentdirections parallel to the sample plane. During data acquisition, thecamera can take images at different translation positions. The imagescaptured at each controlled position can be used to measure a map ofsurface normals, a depth map, and maps of reflectance properties at eachtranslation position and then this data can be stitched together.

H. Wavelength/Polarization Control and Motion Control

In one implementation, a 3DSM system combines wavelength/polarizationcontrol and motion control to capture images at different wavelength,different polarization states, and different controlled positions. Forexample, a 3DSM system may comprise one or more of thewavelength/polarization control devices in FIGS. 27A-27C, FIGS. 28A-28C,and FIGS. 29A-29C and one or more of the motion control devices in FIGS.31A-31C, and FIGS. 32A-32C.

I. Surface Emissivity Properties

In some implementations, a 3DSM system is configured to obtain surfaceemissivity properties. During data acquisition, the engineeredillumination is switched to an on state to illuminate the surface andthen switched to an off state during which the camera receives theemitted light (emissions) from the surface and capture intensitymeasurements based on the received emissions. These 3DSM systems can beconfigured to control one or more of motion of the emissive surface, themotion of the engineered illumination and the camera, the wavelengthand/or the polarization of the emitted light from the sample surface.

FIGS. 33-35 are schematic diagrams depicting operations of a dataacquisition and analysis phase of a 3DSM method that obtains surfaceemissivity properties, according to embodiments. FIG. 33 is a schematicdiagram illustrating a data acquisition phase performed by a 3DSM system3300 configured to control the motion of the emissive surface, accordingto an embodiment. FIG. 34 is a schematic diagram illustrating a dataacquisition phase performed by a 3DSM system 3400 configured to controlthe motion of engineered illumination and a camera, according to anembodiment. FIG. 35 is a schematic diagram illustrating a dataacquisition phase performed by a 3DSM system 3500 configured to controlthe wavelength and/or the polarization of the emitted light from thesample surface, according to an embodiment. In one implementation, themotion control shown in FIG. 33 and FIG. 34 and thewavelength/polarization control shown in FIG. 35 are combined togetherto capture images at different controlled positions, differentwavelength, and different polarization states.

FIG. 33 is a schematic diagram of operations of a data acquisition phasethat controls the motion of an emissive surface 3382 and operations ofan analysis phase that obtains surface emissivity properties, asperformed by a 3DSM system 3300 of an embodiment. The 3DSM system 3300comprises a sample platform for receiving a sample 3380 with an emissivesurface 3382, an engineered illumination system 3330, a motion controldevice 3332 coupled to the sample platform 3340, a camera 3360 forcapturing images, and one or more processors 3322. When the engineeredillumination system 3330 is in the on state, the engineered illuminationsystem 3330 provides engineered illumination to the emissive surface3382 of the sample 3380. The illustrated example shows the illuminationsystem 3330 in the off state after an on state. At this instant,emissive surface 3382 is emitting light (emissions) that is propagatedto the camera 3360. The motion control device 3332 is configured forcontrols such as translation, tilt, and rotation that can be applied tothe emissive surface 3382. At each motion controlled position, imagesare captured under different exposure settings by the camera 3360.Although two exposures, Exposure A and Exposure B, are shown, additionalimages can be taken. During operation, the image data of the capturedimages is communicated in a signal to the one or more processors 3322and the processor(s) use the image data to measure the surfaceemissivity properties, such as color, brightness, and uniformity.

FIG. 34 are diagrams illustrating operations of a data acquisition phasethat controls the motion of engineered illumination and camera andoperations of an analysis phase that obtain surface emissivityproperties, as performed by a 3DSM system 3400 of an embodiment. The3DSM system 3400 comprises an assembly 3401 that including theengineered illumination system 3430 and the camera 3460 for capturingimages. The 3DSM system 3400 further comprises a motion control device3432 coupled to the assembly 3401, and one or more processors 3422. Whenthe engineered illumination system 3430 is in the on state, theengineered illumination system 3430 provides engineered illumination tothe emissive surface 3482 of the sample 3480. The illustrated exampleshows the illumination system 3430 in the off state after an on state.At this instant, emissive surface 3482 is emitting light (emissions)that is propagated to the camera 3460. The motion control device 3432 isconfigured for different motion controls such as translation, tilt, androtation as can be applied to both the engineered illumination and thecamera 3460 or to the camera 3460 alone. At each motion controlledposition, images are captured under different exposure settings by thecamera 3460. Although two exposures, Exposure A and Exposure B, areshown, additional images can be taken. During operation, the image dataof the captured images is communicated in a signal to the one or moreprocessors and the processor(s) use the image data to measure thesurface emissivity properties, such as color, brightness, anduniformity.

FIG. 35 are diagrams illustrating operations of a data acquisition phasethat control the wavelength and/or polarization of emitted light 3598from an emissive surface 3582 of a sample 3580 and operations of ananalysis phase that obtain surface emissivity properties as performed bya 3DSM system 3500, according to an embodiment. The 3DSM system 3500comprises an engineered illumination system 3530, a camera 3560 forcapturing images, a wavelength/polarization control device 3538configured to change the wavelength and/or polarization states of lightemitted from the emissive surface 3582 of the sample 3580, and one ormore processors 3522. When the engineered illumination system 3530 is inthe on state, the engineered illumination system 3530 providesengineered illumination to the emissive surface 3582 of the sample 3580.The illustrated example shows the illumination system 3530 in the offstate after an on state. At this instant, the emissive surface 3582 isemitting light (emissions) and the wavelength/polarization controldevice 3538 is controlling the wavelength and/or polarization state ofthe emissions and propagates light to the camera 3560. The wavelengthand polarization of emitted light can be controlled alone or becontrolled together, and the surface emissivity properties are measuredat different wavelength and/or polarization states by the 3DSM system3500.

J. Surface Translucency Properties

In some implementations, a 3DSM system can be configured to obtainsurface translucency properties. For example, when the surface of asample is found to be translucent, the engineered illumination can beswitched to an off state. A backlight placed behind the translucentsurface is then illuminated, and the camera only measures thetransmitted light from the translucent surface. The intensitymeasurements can be used to determine the surface translucencyproperties of the sample surface.

FIG. 36 is a schematic diagram of operations of a data acquisition phasethat controls the motion of a translucent surface 3682 and operations ofan analysis phase that obtains surface translucency properties, asperformed by a 3DSM system 3600 of an embodiment. The 3DSM system 3600comprises a backlight source 3690, a motion control device 3632, anengineered illumination system 3630, a camera 3660, and one or moreprocessors 3622. The motion control device 3632 is coupled to a sampleplatform shown having the sample 3680 disposed thereon during operation.The illustrated example shows the engineered illumination system 3630 inan off state. At this instant, the backlight source 3690 located behindthe translucent surface 3682 provides backlight 3691. The camera 3660only measures transmitted light 3698 through the translucent surface3682. The motion control device 3632 controls the motion and position ofthe translucent surface 3682. At each of the controlled positions,multiple images are captured under different exposure settings:“Exposure A” and “Exposure B.” Although two exposures are shown,additional exposures may be used in another embodiment. As depicted inFIG. 36, the image data of the captured images is communicated in asignal to the one or more processors 3622 and the processor(s) use theimage data to measure the surface translucency properties, such as, forexample, one or more of color, transmission, and alpha map.

FIG. 37 is a schematic diagram of operations of a data acquisition phasethat controls the motion of the backlight 3791 as performed by a 3DSMsystem 3700 of an embodiment. The 3DSM system 3700 comprises a backlightsource 3790, a motion control device 3732, an engineered illuminationsystem 3730, a camera 3760, and one or more processors 3722. Theillustrated example shows the engineered illumination system 3730 in anoff state. At this instant, the backlight source 3790 located behind thetranslucent surface 3782 provides backlight 3791. The camera 3760 onlymeasures transmitted light 3798 through the translucent surface 3782.The motion control device 3732 controls the motion of the backlight3791. Different motion controls such as, for example, one or more oftranslation, tilt, and rotation can be applied to backlight 3791. Ateach of the controlled positions, multiple images are captured underdifferent exposure settings: “Exposure A” and “Exposure B.” Although twoexposures are shown, additional exposures may be used in anotherembodiment. As depicted in FIG. 37, the image data of the capturedimages is communicated in a signal to the one or more processors 3722and the processor(s) use the image data to measure the surfacetranslucency properties, such as, for example, one or more of color,transmission, and alpha map.

FIG. 38 is a schematic diagram of operations of a data acquisition phasethat controls the motion of the engineered illumination and the camera3860 and operations of an analysis phase that obtains surfacetranslucency properties, as performed by a 3DSM system 3800 of anembodiment. The 3DSM system 3800 comprises, an assembly 3831 comprisingan engineered illumination system 3830 and a camera 3860 for capturingimages, a motion control device 3833 coupled to the assembly 3831, andone or more processors 3822. The illustrated example shows theillumination system 3830 in an off state. At this instant, a backlightsource 3890 located behind the translucent surface 3882 providesbacklight 3891. The camera 3860 only measures transmitted light 3898through the translucent surface 3882. The motion control device 3833controls the motion of engineered illumination and camera. Differentmotion controls such as translation, tilt, and rotation can be appliedto both engineered illumination and the camera 3860 or to the camera3860 alone, and at each controlled position images are captured underdifferent exposures. At each of the controlled positions, multipleimages are captured under different exposure settings: “Exposure A” and“Exposure B.” Although two exposures are shown, additional exposures maybe used in another embodiment. As depicted in FIG. 38, the image data ofthe captured images is communicated in a signal to the one or moreprocessors 3822 and the processor(s) use the image data to measure thesurface translucency properties, such as, for example, one or more ofcolor, transmission, and alpha map.

FIG. 39 is a schematic diagram of operations of a data acquisition phasethat controls the wavelength and/or polarization of both a backlight3938 and/or transmitted light from a surface 3982, as performed by a3DSM system 3900 of an embodiment. The 3DSM system 3900 comprises, abacklight source 3938, a first wavelength/polarization control device3990, an engineered illumination system 3930, a secondwavelength/polarization control device 3939, a camera 3960 for capturingimages, and one or more processors 3922. The illustrated example showsthe engineered illumination system 3930 in an off state. At thisinstant, a backlight source 3938 located behind the translucent surface3982 provides backlight 3992. The camera 3960 only measures transmittedlight 3999 through the translucent surface 3982. The firstwavelength/polarization control device 3990 controls the wavelengthand/or polarization of the backlight 3938 and the secondwavelength/polarization control device 3939 controls the wavelengthand/or polarization of transmitted light 3998 from the surface 3982 ofthe sample 3980. The wavelength and polarization state can be controlledalone or be controlled together. The backlight and transmitted light canalso be controlled individually or be controlled together. At each ofthe controlled positions, multiple images are captured under differentexposure settings: “Exposure A” and “Exposure B.” Although two exposuresare shown, additional exposures may be used in another embodiment. Asdepicted in FIG. 39, the image data of the captured images iscommunicated in a signal to the one or more processors 3922 and theprocessor(s) use the image data to measure the surface translucencyproperties at different wavelength and polarization states.

In one embodiment, the motion control shown in FIGS. 37-38 and thewavelength/polarization control shown in FIG. 39 can also be combinedtogether to capture images at different controlled positions, differentwavelength, and different polarization state.

K. An Example of a 3DSM System with Engineered Illumination in the OffState

FIG. 40 depicts a schematic diagram depicting an implementation of the3DSM system 900 of FIGS. 11A-B and FIG. 12. In this implementation, the3DSM system 900 is configured to switch the engineered illumination toan off state. The 3DSM system 900 comprises an engineered illuminationsystem having optical elements including a diffuser 936, a rotatingmirror 937, and a beam-splitter 938. The engineered illumination systemalso includes collimated illumination 935 from an illumination device(not shown). The 3DSM system 900 further comprises a camera 960. The3DSM system 900 also includes an x-axis, a y-axis, and a z-axis (notshown) orthogonal to the x-axis and the y-axis. The 3DSM system 900 alsoincludes a motion control device 932 coupled to the rotating mirror 937.The motion control device 932 is configured to rotate the mirror 937about the x-axis and the z-axis (two orthogonal rotational directions).The illustrations show the rotating mirror 937 being rotated about thez-axis between a first position (dotted line) and the current secondposition (solid line). In FIG. 40, the diffuser 936 receives collimatedillumination 935 and the collimated illumination is transmitted throughthe diffuser 936. The motion control device 932 turns the rotatingmirror 937 to a position that does not reflect the light from thediffuser 936 to the beam splitter 938, for example, a position that isparallel to the diffuser 936. In the illustrated example, the mirror 937is rotated from a first position to a second position and the beam A2 isreflected back as beam B2 and does not fall on the beam splitter 938. Inthis example, the object surface 982 does not receive any light from theengineered illumination. By switching engineered illumination to thisoff state the camera 960 only measures the emitted light and/ortransmitted light from the surface 982. The images in thisimplementation can be used to measure the surface emissivity propertiesor the surface translucency properties.

V. 3DSM Methods

FIG. 42 is a flowchart depicting operations of a 3DSM method, accordingto various implementations. The 3DSM method is implemented by a 3DSMsystem of various implementations described herein. The 3DSM methodgenerally comprises a data acquisition phase, an analysis phase and anoptional display phase. One or more processors (e.g., processor(s) 22)in FIG. 2) execute instructions stored in memory (e.g., memory 24 inFIG. 2) to perform one or more operations of the analysis phase and ofthe optional display phase.

At operation 4210, the intensity images are acquired for a run of a dataacquisition phase. During the data acquisition phase, engineeredillumination of one or more ray bundles is directed to N differentillumination directions incident a surface of a sample being analyzed.Examples of engineered illumination systems that can be used to generatethe engineered illumination are described in sections above. The Ndifferent illumination directions are typically based on rotating thebundles in two orthogonal directions. For example, the 3DSM system mayrotate the bundles in 20 rotational angles in a first direction and in20 rotational angles along a second direction orthogonal to the firstdirection for a combination of four hundred (400) illuminationdirections. At each illumination direction, n intensity images arecaptured at different exposure settings by a camera. FIG. 43 is aflowchart of sub-operations of operation 4210.

At operation 4212, the 3DSM method initializes the counter i=1. Atoperation 4214, the engineered illumination system provides engineeredillumination to the surface of a sample at an illumination directionθ_(i). For example, the engineered illumination system may provide oneor more ray bundles of illumination beams of varying intensities to thesurface at an illumination direction θ_(i). Different systemconfigurations for generating the engineered illumination are describedin various examples described in sections above. At each illuminationdirection, the sensor(s) of a camera capture n intensity images fordifferent exposure settings (operation 4216). In one case, two (2)intensity images are captured at each illumination direction. In anothercase, three (3) intensity images are captured at each illuminationdirection. In another case, more than three (3) intensity images arecaptured at each illumination direction.

At operation 4218, the 3DSM method increments the counter i=i+1. Atoperation 4220, the 3DSM method returns to repeat operations 4214, 4216,and 4218 until i is greater than N. If i is greater than Nat operation4220, the 3DSM method preforms optional operations (denoted by dashedline) 4222, 4224. If optional operation 4222 is implemented, thewavelength and/or polarization of light is changed to a different stateand the method returns to operation 4212 to repeat the operations 4212,4214, 4216, 4218, and 4220 to capture intensity images for the newwavelength/polarization state for the N illumination directions.Alternatively, the wavelength/polarization can be changed to differentstates at each illumination direction θ_(i) after operation 4214.

Optional operation 4224 is typically implemented where the surface of asample is larger than the camera's field-of-view. During this optionaloperation 4224, the sample is translated to different positions torepeat the operations 4212, 4214, 4216, 4218 and optionally 4222 tocapture intensity images for different portions (e.g., each portionhaving the area of the camera's field of view) of the sample surfaceuntil the entire sample surface is analyzed. In one example, a motioncontrol device (e.g., an x-y stage) is used to translate the sample todifferent locations incrementally so that the camera has a field-of-viewof different portions of the sample surface. In another example, themotion control device can translate the illumination device(s) and/orthe camera relative to the sample.

Returning to FIG. 42, at operation 4300, one or more processors executeinstructions that determine a map of surface normals for each surfacepixel based on the intensity images. FIG. 44 is a flowchart ofsub-operations of operation 4300.

At operation 4310, the 3DSM method generates a measured sensor responsefor each of the sensor pixels based on the intensity images capturedduring the data acquisition phase. Each sensor pixel corresponds to asurface point of the surface. During the data acquisition phase, one ormore sensors of a camera of the 3DSM system capture intensity values ateach sensor pixel for the N different illumination directions and forthe different exposure settings and optionally for differentwavelength/polarization states. A measured sensor response isconstructed for each sensor pixel based on the measured intensity valuesat the sensor pixel for the N different illumination directions for oneof the different exposures. In one case, the 3DSM system determines oneof the different exposures to use in the measured sensor response basedon the best contrast or dynamic range. In another case, all the measuredintensity values under different exposures are converted to a radiancemap or a high dynamic range image.

At operation 4320, the 3DSM method determine a surface normal for eachsensor pixel by matching the measured sensor response to the best fit ofa set of pre-determined sensor responses for different surface normals.Because the illumination is engineered with a known distribution, asensor response can be pre-determined by taking a convolution of theengineered beam distribution function with all the possible surfacenormal directions at different illumination directions. The measuredsurface normal can then be determined based on the best matching betweencaptured sensor response and the pre-determined sensor response. A bestmatched pre-determined sensor response can be found based on differentmetrics, such as correlation and minimum difference. The surface normalvector of the matched pre-determined sensor response is then assigned tothat pixel. The same sensor response matching process is repeated forall the pixels in the captured images to determined surface normals ofthe surface.

In one example, the sensor response matching for surface normalestimation can be described using the following Eqn. 1:

$\begin{matrix}{{o(n)} = {\sum\limits_{i}\;\left\lbrack {S_{i} - S_{i}^{\prime}} \right\rbrack^{2}}} & \left( {{Eqn}.\mspace{14mu} 1} \right)\end{matrix}$Where n is the surface normal, S_(i) is the measured sensor responsewith engineered illumination rotated to the i^(th) direction, and S_(i)′is the predicted sensor response also with engineered illuminationrotated to the i^(th) direction. Other objective functions can also beused. The unknown surface normal n can be estimated based onoptimization or using a brute force approach in which the predictedsensor responses are computed for various surface normal conditions andare compared with measured sensor response at each surface location.

Returning to FIG. 42, once the surface normals are determined for theentire surface, the 3DSM method estimates a depth map to determine ameasured surface topography from the surface normals (operation 4320). Adepth map can be calculated from the determined surface normals. Thisoperation can be done based on an objective function that is theintegral of the square of the errors between surface gradients ofestimated surface depth and measured surface gradients.

At operation 4340, the 3DSM method determines various surface propertiesof the sample based on the surface normals. Reflectance properties, suchas diffuse albedo, specular albedo and roughness can be estimated basedon a BRDF model, such as the Ward model, the Cook-Torrance model and theBlinn-Phong Model.

The BRDF model is a mathematical approximation of how surfaces react tolight, and models the diffuse characteristics and specular highlight ofsurface material. When light is incident on a surface from differentangles, the reflected light represents different BRDF characteristics.Since the engineered illumination is rotated to different orientations,a large number of angular samplings of surface's BRDF are collected. Thecollected data from the images is fitted into a BRDF mathematical modelsuch as, for example, the Ward model, the Cook-Torrance model, or theBlinn-Phong Model. These BRDF models are generally a non-linear functionwith illumination normal and camera viewing normal as the variables, andwith normal, diffuse reflection, specular reflection, and roughness asthe coefficients. Different BRDF models may have different coefficients,such as anisotropic, basecolor, metallic, and glossiness.

For example, an isotropic Ward BRDF model can be described as anon-linear function described in Eqn. 2:ρ(LN,VN)=ƒ(n,ρ _(d),ρ_(s),β)  (Eqn. 2)where LN is the normal of incident light, VN is the normal of cameraviewing direction, n is the surface normal, ρ_(d) is the diffusereflectance coefficient, ρ_(s) is the specular reflectance coefficient,and β is the roughness coefficient. ƒ(n, ρ_(d), ρ_(s), β) is anon-linear function describing the BRDF model. LN and VN are variableswhich can be known based on system's design configuration or based oncalibration. n, ρ_(d), ρ_(s), and β are coefficients that need to beestimated. The surface normals are estimated at operation 4300. Theρ_(d), ρ_(s), and β coefficients can then be estimated using anobjective function as follows:

$\begin{matrix}{{O\left( {\rho_{d},\rho_{s},\beta} \right)} = {\sum\limits_{i}\;\left\lbrack {S_{i} - {\left( {{LN} \cdot {VN}} \right){f\left( {\rho_{d},\rho_{s},\beta} \right)}}} \right\rbrack^{2}}} & \left( {{Eqn}.\mspace{14mu} 3} \right)\end{matrix}$

Other objective functions can also be used. A brute force approach ornon-linear optimization can be applied to estimate the ρ_(d), ρ_(s), andβ coefficients, and this process is repeated for all the surfacelocations. If desired the estimated surface normal at operation 4300 canalso be used as an initial point. The surface normal n can then berefined and ρ_(d), ρ_(s) and β coefficients are estimated using anobjective function as below. Other objective functions can also be usedin other implementations.

$\begin{matrix}{{O\left( {n,\rho_{d},\rho_{s},\beta} \right)} = {\sum\limits_{i}\;\left\lbrack {S_{i} - {\left( {{LN} \cdot {VN}} \right){f\left( {n,\rho_{d},\rho_{s},\beta} \right)}}} \right\rbrack^{2}}} & \left( {{Eqn}.\mspace{14mu} 4} \right)\end{matrix}$

Optionally (denoted by dashed line), at operation 4360, the 3DSM methodcan generate one or more renderings and/or performs additional analyses.Renderings can be generated using techniques such as ray casting or raytracing based on measured surface normal, depth, reflectance properties,emissivity properties, and translucency properties. During the optionaldisplay operation, the renderings and/or other output is provided on adisplay such as the display 70 shown in FIG. 2 (operation 4390).

FIG. 45 is a block diagram depicting inputs and outputs of the renderingoperation 4360, according to an embodiment. In FIG. 45, the illustratesphysically-based rendering and photorealistic rendering using measuredsurface normal, depth, reflectance properties, surface emissivityproperties, and surface translucency properties. As shown, the measuredsurface normal, depth, and reflectance properties such as diffusereflection, specular reflection, glossiness, base color, metallic map,and roughness can be used to render the reflective appearance of anobject. The measured surface emissive properties such as color,brightness, and uniformity can be used to render the emissive appearanceof an object. The measured surface translucency properties such ascolor, transmission, and alpha map can be used to render the translucentappearance of an object.

FIGS. 46A-46H are illustrations of surface reflectance properties,surface normal, and depth map of a synthetic leather sample measured bya 3DSM system, according to an implementation. FIG. 46A is a measureddiffuse reflection of the synthetic leather surface. FIG. 46B is ameasured specular reflection of the synthetic leather surface. FIG. 46Cis a measured glossiness of the synthetic leather surface. FIG. 46D is ameasured base color that describes the combined color appearance of bothdiffuse reflection and specular reflection of the synthetic leathersurface. FIG. 46E is a measured metallic map that describes thedielectric and metallic properties of the synthetic leather surface.FIG. 46F is a measured roughness of the synthetic leather surface. FIG.46G is a measured normal map of the synthetic leather surface. FIG. 4611is a measured depth map of the synthetic leather surface. FIG. 46I is aphysically-based rendering of the synthetic leather using measuredparameters.

Defect Detection

In certain implementations, the 3DSM method can detect defects based oncertain results determined by a 3DSM method such as described withrespect to FIGS. 42-44. FIG. 47 is a block diagram illustratingoperations of a 3DSM method that include a defect detection process,according to an embodiment. In FIG. 47, the diagram illustrates surfacedefect visualization using measured surface normal, depth, reflectanceproperties, surface emissivity properties, and surface translucencyproperties. The defects on a reflective surface may present differentnormal, depth and reflectance properties. Similarly defects on anemissive surface may present different emissivity properties and defectson a translucent surface may present different translucency properties.3D rendering based on measured reflectance properties, measured emissiveproperties, and measured translucency properties can be used to providedefect visualization in a three-dimensional space. Different lightingconditions and different viewing conditions can be virtually changed inthe 3D rendering.

FIG. 48 is a block diagram illustrating a defect detection process,according to an embodiment. In FIG. 48, the diagram illustrates surfacedefect detection and classification using measured surface normal,depth, reflectance properties, surface emissivity properties, andsurface translucency properties. Image processing and patternrecognition techniques can be applied to the normal, depth andreflectance properties measured on a reflective surface, be applied tothe emissivity properties measured on an emissive surface, and beapplied to the translucency properties measured on a translucentsurface. Any defect that presents different appearance on these measuredparameters can be detected. The detected defects can then be passed to amachine learning module. Based on a surface defect library that haslearned possible defect features surface defects can be classified intodifferent categories.

FIGS. 49A-49F are illustrations of measured surface reflectanceproperties, normal, and depth of a LCD panel that is contaminated withdirt and fiber as measured by a 3DSM system, according to animplementation. FIG. 49A is an illustration of measured specularreflection. FIG. 49B is an illustration of measured diffuse reflection.FIG. 49C is an illustration of measured roughness. FIG. 49D is anillustration of a measured normal map. FIG. 49E is an illustration of ameasured depth map. The dirt and fiber's reflection is more diffusivethan the LCD panel's reflection, and therefore dirt and fiber areshowing as darker in the specular map in FIG. 49A, but are much brighterin the diffuse map in FIG. 49B. It can also be seen that dirt and fibershow much larger roughness values in roughness map. In the normal mapand depth map it can be seen that dirt and fiber have distinguishedvalues, which indicate different height and 3D shape. All these measuredparameters can be used to identify the dirt and fiber defects on the LCDpanel. FIG. 49F is an illustration of surface defect visualization usingmeasured parameters. The measured normal, depth, and reflectanceproperties are used in the 3D rendering. A virtual light is placed inthe scene and a camera is virtually placed at an oblique angle. Thedefects can be clearly visualized in the 3D rendering shown in FIG. 49F.

FIGS. 50A-50F are illustrations of measured surface reflectanceproperties, normal, and depth of a LCD panel with scratches and pits asmeasured by a 3DSM system, according to an implementation. FIG. 50A isan illustration of measured specular reflection. FIG. 50B is anillustration of measured diffuse reflection. FIG. 50C is an illustrationof measured roughness. FIG. 50D is an illustration of measured normalmap. FIG. 50E is an illustration of a measured depth map. FIG. 50F is anillustration of surface defect detection using measured parameters. Thepixels in red are detected defects.

FIGS. 51-52 are illustrations depicting a surface defect detectionprocess using measured surface emissive properties of a LCD panel thathas white spot and a LCD panel that has a non-uniformity as measured bya 3DSM system, according to an implementation. The images were measuredby switching the engineered illumination to an off state. FIG. 51 is ameasured brightness map of a LCD panel that is with white spot, which isidentified within the circle. FIG. 52 is a measured uniformity map of aLCD panel that is not uniform. A non-uniform region is identified withinthe ellipse.

In one embodiment, a 3DSM method uses a calibration process by using amirror and a chrome sphere to compensate for imperfect system alignmentand provide more accurate illumination direction information.

In one embodiment, addition multiple illuminations can be added atoblique angles to improve the estimation.

In one embodiment, system alignment can be performed by using a mirroras a reference object.

In one embodiment, wavelength and polarization control can be applied tothe engineered illumination and reflected light from object surface.

In one embodiment, motion control can be applied to the object surface,engineered illumination and camera.

In one embodiment, the emissivity properties of a emissive surface canbe measured by switching the engineered illumination to an off state.

In one embodiment, the translucency properties of a translucent surfacecan be measured by switching the engineered illumination to an off stateand by placing a backlight behind the surface.

In one embodiment, measured surface normal, depth, reflectanceproperties, surface emissivity properties, and surface translucencyproperties can be used for physically-based rendering and photorealisticrendering.

In one embodiment, measured surface normal, depth, reflectanceproperties, surface emissivity properties, and surface translucencyproperties can be used for surface defect visualization, surface defectdetection and classification.

Modifications, additions, or omissions may be made to any of theabove-described embodiments without departing from the scope of thedisclosure. Any of the embodiments described above may include more,fewer, or other features without departing from the scope of thedisclosure. Additionally, the steps of the described features may beperformed in any suitable order without departing from the scope of thedisclosure.

It should be understood that the present invention as described abovecan be implemented in the form of control logic using computer softwarein a modular or integrated manner. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will know andappreciate other ways and/or methods to implement the present inventionusing hardware and a combination of hardware and software.

Any of the software components or functions described in thisapplication, may be implemented as software code to be executed by aprocessor using any suitable computer language such as, for example,Java, C++ or Perl using, for example, conventional or object-orientedtechniques. The software code may be stored as a series of instructions,or commands on a CRM such as a random access memory (RAM), a read onlymemory (ROM), a magnetic medium such as a hard-drive or a floppy disk,or an optical medium such as a CD-ROM. Any such CRM may reside on orwithin a single computational apparatus, and may be present on or withindifferent computational apparatuses within a system or network.

Although the foregoing disclosed embodiments have been described in somedetail to facilitate understanding, the described embodiments are to beconsidered illustrative and not limiting. It will be apparent to one ofordinary skill in the art that certain changes and modifications can bepracticed within the scope of the appended claims.

One or more features from any embodiment may be combined with one ormore features of any other embodiment without departing from the scopeof the disclosure. Further, modifications, additions, or omissions maybe made to any embodiment without departing from the scope of thedisclosure. The components of any embodiment may be integrated orseparated according to particular needs without departing from the scopeof the disclosure.

What is claimed is:
 1. A 3D surface measurement system comprising: anengineered illumination system configured to provide at least one raybundle sequentially at N illumination directions incident a surface of asample being imaged, wherein each ray bundle comprises illuminationbeams of various intensities, the illumination beams of each ray bundleconfigured in different directions converging toward the surface of thesample during operation; a first camera with an imaging lens and atleast one sensor configured to acquire intensity images based on lightreceived from the illuminated sample, each intensity image acquiredwhile the at least one ray bundle is directed at one of the Nillumination directions; and a controller configured to executeinstructions to: determine a sensor response at each sensor pixel of aplurality of sensor pixels of the at least one sensor, the sensorresponse determined from the intensity images corresponding to the Nillumination directions; match the sensor response at each sensor pixelto one of a plurality of predetermined sensor responses to determine asurface normal at each sensor pixel; and construct a map of surfacenormals of the surface of the sample by combining the determined surfacenormals of the plurality of sensor pixels of the at least one sensor. 2.The 3D surface measurement system of claim 1, wherein the engineeredillumination system comprises: an illumination device with at least onelight source; and one or more optical elements configured to generateand propagate the at least one ray bundle based on light from the atleast one light source to the surface of the sample and to propagatelight from the illuminated surface to the first camera.
 3. The 3Dsurface measurement system of claim 2, wherein the one or more opticalelements comprise a diffuser configured to spread illumination from theat least one light source to generate the illumination beams of variousintensities of each ray bundle according to a transmission profile ofthe diffuser.
 4. The 3D surface measurement system of claim 3, whereinthe diffuser transmission profile is configured to spread theillumination beams of various intensities to different angles accordingto a Gaussian or Gaussian-like distribution.
 5. The 3D surfacemeasurement system of claim 2, wherein the one or more optical elementsare further configured to distribute the illumination beams of variousintensities at different angles according to a linear gradient.
 6. The3D surface measurement system of claim 2, wherein the one or moreoptical elements are further configured to distribute the illuminationbeams of various intensities at different angles according to a randomdistribution.
 7. The 3D surface measurement system of claim 2, furthercomprising an additional illumination source configured to directillumination incident the surface of the sample at an oblique angle. 8.The 3D surface measurement system of claim 2, wherein the one or moreoptical elements comprise: a mirror configured to receive illuminationbeams propagated from the at least one light source; and a beam splitterconfigured to receive the illumination beams from the mirror andreflecting the illumination beams to the surface of the sample andconfigured to transmit light reflected from the surface of the sample.9. The 3D surface measurement system of claim 2, wherein the one or moreoptical elements comprise: a combination filter of a plurality ofneutral density filters, the combination filter configured to engineerillumination from the at least one light source into the illuminationbeams of various intensities of each of the at least one ray bundle,each neutral density filter associated with one illumination beam; and acollimator configured to collimate the illumination beams propagatedfrom the combination filter.
 10. The 3D surface measurement system ofclaim 1, wherein the engineered illumination system comprises: anillumination array having a plurality of illumination elements, theillumination array configured to activate one of the illuminationelements at a single exposure time; and a collimator configured tocollimate illumination received from the one activated illuminationelement; and a diffuser configured to spread the collimated illuminationfrom the collimator to the illumination beams of various intensities ofeach ray bundle according to a transmission profile of the diffuser. 11.The 3D surface measurement system of claim 1, further comprising amotion control device configured to manipulate the sample and/or theengineered illumination and the camera together.
 12. The 3D surfacemeasurement system of claim 11, wherein the motion control device isconfigured to manipulate the sample and/or the engineered illuminationand the camera together to shift the illumination beams to differentregions of the surface of the sample.
 13. The 3D surface measurementsystem of claim 11, wherein the motion control device is configured tomanipulate the sample and/or the engineered illumination and the cameratogether to direct the illumination beams in the N illuminationdirections.
 14. The 3D surface measurement system of claim 2, furthercomprising a wavelength and/or polarization control device configured toalter the wavelength and/or polarization of the at least one ray bundleor alter the wavelength and/or polarization of reflected light from thesurface of the sample.
 15. The 3D surface measurement system of claim 2,further comprising a first wavelength and/or polarization control deviceconfigured to alter the wavelength and/or polarization of the at leastone ray bundle and a second wavelength and/or polarization controldevice configured to alter the wavelength and/or polarization of thereflected light from the surface of the sample.
 16. The 3D surfacemeasurement system of claim 2, wherein the controller further configuredto execute instructions to determine a depth map and/or surfaceproperties of the surface of the sample.
 17. The 3D surface measurementsystem of claim 2, further comprising a second camera with an imaginglens and at least one sensor, the second camera configured to captureintensity images at the N illumination directions based on lightreceived from the illuminated sample, wherein the second camera capturesintensity images from a different angle than the first camera.
 18. The3D surface measurement system of claim 1, wherein the illumination beamsof each ray bundle converge to a point.
 19. A 3D surface measurementmethod comprising: receiving a plurality of intensity images of a samplein a signal from at least one sensor of a camera, the plurality ofintensity images captured at a plurality of exposure times, eachintensity image acquired while one or more ray bundles is directed atone of N illumination directions, each ray bundle comprisingillumination beams of various intensities, the illumination beams ofeach ray bundle directed in different directions converging toward thesurface of the sample; determining a sensor response at each sensorpixel of a plurality of sensor pixels of the at least one sensor, thesensor response determined from the intensity images at the Nillumination directions; matching a sensor response at each sensor pixelto one of a plurality of predetermined sensor responses to determine asurface normal at each sensor pixel; and construct a map of surfacenormals of the surface of the sample by combining the determined surfacenormals of the plurality of sensor pixels of the at least one sensor.20. The 3D surface measurement method of claim 19, further comprisingdetermining a depth map of the surface from the map of surface normals.21. The 3D surface measurement method of claim 19, further comprisingdetermining a map of surface properties of the surface based on the mapof surface normals.
 22. The 3D surface measurement method of claim 21,wherein the surface properties comprise one or more of a reflectanceproperty, an emissivity property, and a translucency property.
 23. The3D surface measurement method of claim 19, further comprisingdetermining a defect in the surface of the sample based a map of surfaceproperties and/or a depth map of the surface, wherein each map isdetermined from the map of surface normals.
 24. The 3D surfacemeasurement method of claim 19, further comprising: determining a depthmap of the surface from the map of surface normals; determining one ormore maps of surface properties of the surface based on the map ofsurface normals; and generating a physically-based rendering based onthe map of surface normals, the depth map, and the at least one of theone or more maps of surface properties of the surface.
 25. A 3D surfacemeasurement method comprising: a) engineering at least one ray bundledirected sequentially to N illumination directions incident a surface asample being imaged, each ray bundle comprising illumination beams ofvarious intensities, the illumination beams of each ray bundle directedin different directions converging toward the surface of the sample; b)acquiring, using at least one sensor of a camera, intensity images at aplurality of exposure times based on light from the illuminated sample,each intensity image acquired while the one or more ray bundles isdirected at one of the N illumination directions; and c) communicatingthe intensity images to one or more processors, wherein the one or moreprocessors determine a sensor response at each sensor pixel of aplurality of sensor pixels of the at least one sensor, the sensorresponse determined from the intensity images at the N illuminationdirections, match the sensor response at each sensor pixel to one of aplurality of predetermined sensor responses to determine a surfacenormal at each sensor pixel, and construct a map of surface normals ofthe surface of the sample by combining the determined surface normals ofthe plurality of sensor pixels of the at least one sensor.
 26. The 3Dsurface measurement method of claim 24, further comprising changingwavelength or polarization of the illumination beams of the at least oneray bundle and/or light from the illuminated sample, and then repeatinga) and b).